Closed huanglianghua closed 4 years ago
Thanks for your question. It is indeed one common concern from the readers. Let me try to explain our perspective here.
Because CIFAR-10 and PTB are small, these datasets are treated as toy-examples in NAS domain. One should firstly demonstrate its method's effectiveness on these easy problems, then generalizes to larger datasets like ImageNet. If these methods cannot solve a toy-example, how could we believe the algorithm can solve a large scale search problem?
In our humble opinion, it is definitely interesting to find one good architecture, which obtains good performances on proxy tasks, by NAS, however, it is equally important to know why and how a NAS algorithm could do. Before our and other concurrent works (e.g. RandomNAS by Li and Talwalker), people treat different components (search space, sampling policy, evaluation from scratch and etc.) as a black box. What we conclude is, through a simple comparison with random search, NAS's success is credited more to the good design of search space rather than the improvement of these algorithms.
Hope it could clarify our motivation and answer your question. If you have further question, please do not hesitate to reply.
If you have any further discussion, please feel free to re-open this issue. I am closing it now.
Hi, as described in the title, I wonder if CIFAR and PTB datasets are too small to support your conclusions in your paper.