The search procedure is slow

[ganji15]

Hi @D-X-Y , thanks for sharing your excellent work. I found your work is very interesting, and I want to try your code.

However, I found the search is not as fast as the paper said. Specifically, I run the command "CUDA_VISIBLE_DEVICES=0 bash ./scripts-cnn/train-cifar.sh GDAS_F1 cifar10 cut" on the PC with a Nvidia TITAN X (Pascal) 12GB, and it eventually takes 27 hours to find a network with the top-1 error rate of 3.31. The following are some logs:

 train[2019-05-13-06:15:55] Epoch: [486][000/521] Time 0.62 (0.62) Data 0.26 (0.26) Loss 0.033 (0.033)  Prec@1 100.00 (100.00) Prec@5 100.00 (100.00)
 train[2019-05-13-06:16:31] Epoch: [486][100/521] Time 0.36 (0.37) Data 0.00 (0.00) Loss 0.129 (0.083)  Prec@1 96.88 (98.62) Prec@5 100.00 (100.00)
 train[2019-05-13-06:17:08] Epoch: [486][200/521] Time 0.38 (0.37) Data 0.00 (0.00) Loss 0.142 (0.084)  Prec@1 95.83 (98.57) Prec@5 100.00 (100.00)
 train[2019-05-13-06:17:45] Epoch: [486][300/521] Time 0.38 (0.37) Data 0.00 (0.00) Loss 0.024 (0.084)  Prec@1 100.00 (98.58) Prec@5 100.00 (100.00)
 train[2019-05-13-06:18:22] Epoch: [486][400/521] Time 0.37 (0.37) Data 0.00 (0.00) Loss 0.057 (0.085)  Prec@1 100.00 (98.54) Prec@5 100.00 (99.99)
 train[2019-05-13-06:19:00] Epoch: [486][500/521] Time 0.36 (0.37) Data 0.00 (0.00) Loss 0.098 (0.087)  Prec@1 95.83 (98.46) Prec@5 100.00 (99.99)
 train[2019-05-13-06:19:07] Epoch: [486][520/521] Time 0.32 (0.37) Data 0.00 (0.00) Loss 0.084 (0.087)  Prec@1 98.75 (98.45) Prec@5 100.00 (99.99)
[2019-05-13-06:19:07] **train** Prec@1 98.45 Prec@5 99.99 Error@1 1.55 Error@5 0.01 Loss:0.087
 test [2019-05-13-06:19:07] Epoch: [486][000/105] Time 0.34 (0.34) Data 0.26 (0.26) Loss 0.104 (0.104)  Prec@1 96.88 (96.88) Prec@5 100.00 (100.00)
 test [2019-05-13-06:19:16] Epoch: [486][100/105] Time 0.09 (0.09) Data 0.00 (0.00) Loss 0.032 (0.139)  Prec@1 98.96 (96.49) Prec@5 100.00 (99.94)
 test [2019-05-13-06:19:16] Epoch: [486][104/105] Time 0.03 (0.09) Data 0.00 (0.00) Loss 0.048 (0.140)  Prec@1 93.75 (96.44) Prec@5 100.00 (99.93)
[2019-05-13-06:19:16] **test** Prec@1 96.44 Prec@5 99.93 Error@1 3.56 Error@5 0.07 Loss:0.140
----> Best Accuracy : Acc@1=96.69, Acc@5=99.93, Error@1=3.31, Error@5=0.07
----> Save into ./output/NAS-CNN/GDAS_F1-cifar10-cut-E600/seed-166-checkpoint-cifar10-model.pth

==>>[2019-05-13-06:19:17] [Epoch=487/600] [Need: 06:21:40] LR=0.0022 ~ 0.0022, Batch=96
 train[2019-05-13-06:19:17] Epoch: [487][000/521] Time 0.69 (0.69) Data 0.31 (0.31) Loss 0.071 (0.071)  Prec@1 98.96 (98.96) Prec@5 100.00 (100.00)
 train[2019-05-13-06:19:54] Epoch: [487][100/521] Time 0.37 (0.37) Data 0.00 (0.00) Loss 0.069 (0.087)  Prec@1 97.92 (98.39) Prec@5 100.00 (99.99)
 train[2019-05-13-06:20:31] Epoch: [487][200/521] Time 0.36 (0.37) Data 0.00 (0.00) Loss 0.125 (0.086)  Prec@1 96.88 (98.45) Prec@5 100.00 (99.99)
 train[2019-05-13-06:21:08] Epoch: [487][300/521] Time 0.37 (0.37) Data 0.00 (0.00) Loss 0.212 (0.086)  Prec@1 94.79 (98.49) Prec@5 100.00 (99.99)
 train[2019-05-13-06:21:45] Epoch: [487][400/521] Time 0.36 (0.37) Data 0.00 (0.00) Loss 0.090 (0.087)  Prec@1 98.96 (98.49) Prec@5 100.00 (99.99)
 train[2019-05-13-06:22:22] Epoch: [487][500/521] Time 0.38 (0.37) Data 0.00 (0.00) Loss 0.150 (0.088)  Prec@1 96.88 (98.44) Prec@5 100.00 (99.99)
 train[2019-05-13-06:22:29] Epoch: [487][520/521] Time 0.31 (0.37) Data 0.00 (0.00) Loss 0.078 (0.088)  Prec@1 97.50 (98.46) Prec@5 100.00 (99.99)
[2019-05-13-06:22:29] **train** Prec@1 98.46 Prec@5 99.99 Error@1 1.54 Error@5 0.01 Loss:0.088
 test [2019-05-13-06:22:29] Epoch: [487][000/105] Time 0.32 (0.32) Data 0.24 (0.24) Loss 0.059 (0.059)  Prec@1 96.88 (96.88) Prec@5 100.00 (100.00)
 test [2019-05-13-06:22:38] Epoch: [487][100/105] Time 0.09 (0.09) Data 0.00 (0.00) Loss 0.029 (0.141)  Prec@1 98.96 (96.42) Prec@5 100.00 (99.93)
 test [2019-05-13-06:22:38] Epoch: [487][104/105] Time 0.03 (0.09) Data 0.00 (0.00) Loss 0.001 (0.140)  Prec@1 100.00 (96.43) Prec@5 100.00 (99.93)
[2019-05-13-06:22:38] **test** Prec@1 96.43 Prec@5 99.93 Error@1 3.57 Error@5 0.07 Loss:0.140
----> Best Accuracy : Acc@1=96.69, Acc@5=99.93, Error@1=3.31, Error@5=0.07
----> Save into ./output/NAS-CNN/GDAS_F1-cifar10-cut-E600/seed-166-checkpoint-cifar10-model.pth

Since I am fresh to NAS,  I cannot figure out what I have missed? 
Any suggestions/help are greatly appreciated! Thanks.

[D-X-Y]

Thanks for your interest. The above script is used to train the searched model, GDAS_F1 indicates one of our searched CNN. We did not include the searching codes in this repo since an extension version is under review.

[ganji15]

@D-X-Y So, the script is only to train a searched model rather than to search a model? (Sigh~).

[D-X-Y]

Yes, it is mainly used to compare with other NAS searched models, since different papers have different hyper-parameters. Sorry for the inconvenience. If you have any questions w.r.t. the searching procedure, I'm glad to help solve your problems.

[ganji15]

@D-X-Y I am glad to hear that. Anyway, thanks for your kind help.

[D-X-Y]

@ganji15 You are welcome~

[Catosine]

Yes, it is mainly used to compare with other NAS searched models, since different papers have different hyper-parameters. Sorry for the inconvenience. If you have any questions w.r.t. the searching procedure, I'm glad to help solve your problems.

I was wondering whether you are going to release codes for model searching. If so, when are you going to do so?

Thanks

[D-X-Y]

@Catosine We plan to release the searching codes upon the acceptance of our extension version. Thanks for your comprehension.

[Randylcy]

There still isn't searching model code in this repo, I wanna know when will U release,? Thx.

[D-X-Y]

@Randylcy We plan to release the searching codes upon the acceptance of our extension version. Thanks for your comprehension. And we would like to let you know as soon as when we release.

[D-X-Y]

@Randylcy Please find searching scripts here https://github.com/D-X-Y/NAS-Projects#usage-2