search

kcyu2014 / eval-nas

PyTorch Code for "Evaluating the search phase of Neural Architecture Search" @ ICLR 2020
MIT License
49 stars 7 forks source link

Clarification of Paragraph in Paper #3

Open theevann opened 4 years ago

theevann commented 4 years ago

Section 4.2 p7, I read:

NAO seems to constantly outperform random search in the reduced space. Nevertheless, the final architecture chosen by NAO is always one of the architectures from the initial pool, which were sampled uniformly randomly. This indicates that the ranking of NAO is not correctly updated throughout the search and that, in practice, in a reduced space, NAO is similar to random search.

I do not understand this paragraph, in particular the second sentence and why you claim NAO is similar to random while seemingly outperforming it. Could you help clarify it ?

kcyu2014 commented 3 years ago

Sorry for my late reply. If you check the standard deviation, you could see that the confidence interval heavily overlaps. So from pure statistical point of view, the performance of random search and NAO is not really different.