search

automl / NASLib

NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.
Apache License 2.0
519 stars 117 forks source link

Extra .json files for zc_intro.py #178

Closed AnaRisnoveanu closed 11 months ago

AnaRisnoveanu commented 11 months ago

I am running tutorial/zc_intro.py and it returns this error: `[Errno 2] No such file or directory: '/home.../naslib/data/zc_nasbench201.json'

File "/home.../naslib/utils/get_dataset_api.py", line 24, in get_zc_benchmark_api with open(datafile_path) as f: File "/home.../tutorial/zc_intro.py", line 113, in zc_api = get_zc_benchmark_api('nasbench201', 'cifar10')

FileNotFoundError: [Errno 2] No such file or directory: '/home.../naslib/data/zc_nasbench201.json'`

I have already checked the data for the benchmark but this file was not there to download.

abhash-er commented 11 months ago

Hi @AnaRisnoveanu,

You could run the following bash script to download the missing json file:

bash scripts/zc/bash_scripts/download_nbs_zero.sh nb201

You could also pass the arguments nb101, nb301, and tnb101 to this bash script for the respective search spaces.

Alternatively, you can also download using these drive links: NB101 NB201 NB301 TNB101-Macro TNB101-Micro

We will soon update the readme about it.

AnaRisnoveanu commented 11 months ago

Thank you, zc_intro.py works fine now NAS-Benchmark-201.

When I set it to NAS-Benchmark-101, I get an error in score = zc_api[str(spec)][pred]['score'] on line 118. There is no str(spec) in the keys of _zcapi.

Exception has occurred: KeyError '(0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 2, 3, 3, 1)' File "/home/anamariar/topush/experimental.nas_ebs/tutorial/zc_intro.py", line 118, in <module> score = zc_api[str(spec)][pred]['score'] KeyError: '(0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 2, 3, 3, 1)'

abhash-er commented 11 months ago

There are 423624 total architectures available in NASBench101 with CIFAR10, but we only have a list of around 10000 in the NASBench101 zero-cost benchmark. That's the reason you are getting this error.

The search spaces having an exhaustive list of the zero-cost benchmarks are only NasBench201, TNB101 Micro, and TNB101 Macro.

You may also refer to the NAS-Bench-Suite-Zero paper for exact numbers and read more about the reports regarding using zero-cost proxies with NAS.