search

skhu101 / GM-NAS

Code for our ICLR'2022 paper "Generalizing Few-Shot NAS with Gradient Matching"
MIT License
21 stars 2 forks source link

Can't get the performance reported in the paper #3

Closed JeiminJeon closed 2 years ago

JeiminJeon commented 2 years ago

Hello,

I'm trying to run the provided code but the results are inconsistent with the paper.

NASBench-201 space on CIFAR-10, after running train_search.py, With DARTS: I get train: 86.59 and test: 90.44 With SNAS: I get valid: 89.38 and test: 92.76.

Both experiments are conducted on seed = 0.

Could you please provide some information about the version (python, pytorch,..) or some other possible solutions? Thank you.

ruocwang commented 2 years ago

Hi,

I think most likely you are taking the results from the last supernet that is printed.

GM-NAS produces N sub-supernets, and each supernet selects an architecture. For NB201, we pick the best out of N architectures, for larger spaces, we use successive halving (see Appendix for details). So if you take a look at the best architecture from the N supernets (separated by ====), that should be able to match the result from the paper.

I was just aware that we mistakenly logged one of the supernets as "Final". Sorry for the confusion! I'll correct the logging.

JeiminJeon commented 2 years ago

Thank you for your response! The problem is solved. Thank you for the great work again!