search

automl / nasbench-1shot1

Apache License 2.0
67 stars 14 forks source link

How to get Spearman correlation in Fig.3 #5

Closed aoiang closed 4 years ago

aoiang commented 4 years ago

Hi there,

Fig3 shows the correlation between the one-shot validation error and the corresponding NAS-Bench-101 test error for each search space. Would you please let me know where can I get the one-shot validation error in your code? Thank you.

JulienSiems commented 4 years ago

Hey @aoiang We are currently working on a restructured version of this repo. The script https://github.com/automl/nasbench-1shot1/blob/master/nasbench_analysis/evaluate_one_shot_models.py calculates the correlation between the one-shot models and the corresponding nasbench-models for each search space. To calculate the correlation we need to evaluate every architecture in the search spaces, by setting the architectural weights accordingly in the saved checkpoint from the epoch we are interested in. Hence the one-shot validation loss is calculated in this script to compute the overall correlation.