search

ruocwang / darts-pt

[ICLR2021 Outstanding Paper] Rethinking Architecture Selection in Differentiable NAS
Apache License 2.0
102 stars 14 forks source link

The accuracy on cifar10 on NAS-Bench-201 #6

Open zhengjian2322 opened 2 years ago

zhengjian2322 commented 2 years ago

Thanks for your work! I ran DARTS-PT on NAS-Bench-201(cifar10) with darts-201.sh and darts-proj-201.sh, but I didn't get the hight accuracy in the paper. The accuracy I got was 85.03. Is this result normal? If not, can you give me some advice?

ruocwang commented 2 years ago

Hi zhengjian, sorry for the late reply as I needed some time to rerun the exp you mentioned. I reran the darts-201 experiment using the current repo with 2 seeds (0, 1) following the commands below:

bash darts-201.sh --seed {X}
bash darts-proj-201.sh --resume_epoch 100 --resume_expid search-darts-201-{X} --seed {X}

where X is the seed number. I got 88.21% and 88.18% test accuracy respectively. Let me know if it aligns with your commands. I did not observe an 85-ish accuracy while I was working on the manuscript or doing the sanity check for the codebase. That been said, I think technically one could not rule out the possibility of getting something higher or lower than 88%, due to the randomness in the search algorithm & database dev.