Reproduce NAS-Bench-201 REA algo

[JackFram]

Appreciated for your great work!

Which Algorithm REA for NAS-Bench-201

Describe the Question when running bash ./scripts-search/NAS-Bench-201-algos/BOHB.sh cifar10 -1 the error

Traceback (most recent call last):
  File "./exps/NAS-Bench-201-algos/reinforce.py", line 290, in <module>
    save_dir, index = main(args, nas_bench)
  File "./exps/NAS-Bench-201-algos/reinforce.py", line 187, in main
    reward, cost_time = train_and_eval(arch, nas_bench, extra_info, dataname)
  File "/home/ubuntu/zzh/XAutoDL/exps/NAS-Bench-201-algos/R_EA.py", line 111, in train_and_eval
    raise ValueError("NOT IMPLEMENT YET")
ValueError: NOT IMPLEMENT YET

occurred, any workaround?

[D-X-Y]

It is because that you did not download the benchmark file. See here, you need to download NAS-Bench-201-v1_1-096897.pth and put it into ${TORCH_HOME}.

In addition, the NAS-Bench-201 has been upgraded to NATS-Bench, and it is suggested to use the BOHB script in NATS-Bench: https://github.com/D-X-Y/AutoDL-Projects/blob/main/exps/NATS-algos/bohb.py#L8 In short, three key difference for your target experiment are:

• NATS-Bench contains more results of each architecture candidate
• Optimized API -- faster and more robust
• Better tuned hyperparameters for BOHB as well as other baselines.

[JackFram]

Cool, my mistake, I ran REA instead of BOHB and there is a line in the code indicating not implemented yet. Anyway, during the traceback, it seems still to be a nas bench API location problem. Appreciated!

[D-X-Y]

No problem~ I should print more informative debug info.