Experiments on NAS

[taoyang1122]

Hi, I found the experiments on neural architecture search very interesting. I try to follow the experimental setting but do it on NAS-Bench-201. I use the Cifar10_ConvNet_10ipc synthetic data, train each network 200 epochs without data augmentation, learning rate = 0.01 and decay by 10 at 100epoch. Surprisingly, I found the validation accuracy is negtively correlated with the grouth truth accuracy (Kendall's tau is around -0.3). Have you ever tried to do the experiments on NAS benchmarks? Does this imply the ConvNet generated synthetic data cannot generalize well to various network structure? Thanks very much for your help.

[PatrickZH]

Thank you for this question! As you known, we implemented NAS experiment on ConvNets. We haven't implemented any experiments on NAS-Bench-201 benchmark. It may require different training setup, e.g. architectures and hyper-parameters, for learning proxy (synthetic) set for NAS-Bench-201 benchmark. In addition, measuring the performance correlation between two sets is meaningful for those selected top-performing architectures, for example top 30 architectures. The main reasons are that 1) we are only interested in top-performing architectures; 2) performance and thus performance correlation of inferior architectures are instable. We will do experiment on NAS-Bench-201 benchmark in our future work.

[PatrickZH]

Hi, I tried the experiments on NAS-Bench-201. The reason why learned synthetic set is not good at ranking these architectures is that this NAS-Bench-201 is not a good benchmark for evaluating synthetic sets. We find that most of architectures in NAS-Bench-201 are crowded in a narrow performance range (within 5% accuracy range). It is too challenging even impossible to rank thousands of architectures with very close performance due to the randomness of network training.