search

kcyu2014 / eval-nas

PyTorch Code for "Evaluating the search phase of Neural Architecture Search" @ ICLR 2020
MIT License
49 stars 7 forks source link

Wouldn't CIFAR and PTB datasets too small to support your conclusions in the paper? #2

Closed huanglianghua closed 4 years ago

huanglianghua commented 4 years ago

Hi, as described in the title, I wonder if CIFAR and PTB datasets are too small to support your conclusions in your paper.

kcyu2014 commented 4 years ago

Thanks for your question. It is indeed one common concern from the readers. Let me try to explain our perspective here.

Because CIFAR-10 and PTB are small, these datasets are treated as toy-examples in NAS domain. One should firstly demonstrate its method's effectiveness on these easy problems, then generalizes to larger datasets like ImageNet. If these methods cannot solve a toy-example, how could we believe the algorithm can solve a large scale search problem?

In our humble opinion, it is definitely interesting to find one good architecture, which obtains good performances on proxy tasks, by NAS, however, it is equally important to know why and how a NAS algorithm could do. Before our and other concurrent works (e.g. RandomNAS by Li and Talwalker), people treat different components (search space, sampling policy, evaluation from scratch and etc.) as a black box. What we conclude is, through a simple comparison with random search, NAS's success is credited more to the good design of search space rather than the improvement of these algorithms.

Hope it could clarify our motivation and answer your question. If you have further question, please do not hesitate to reply.

kcyu2014 commented 4 years ago

If you have any further discussion, please feel free to re-open this issue. I am closing it now.