automl / NASLib

NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.
Apache License 2.0
528 stars 117 forks source link

Recreate a subset of benchmarks #167

Open svmhkk opened 1 year ago

svmhkk commented 1 year ago

Hi, what would be the best way to recreate a subset of benchmarking accuracy data from the ZC json? For ex: I would want to replicate the val accuracy obtained for the architecture 1-100 indexed from NASBench201. I understand that the val accuracies and corresponding ZC scores were open-sourced to remove the need for training, but I need to perform training for an internal evaluation. How do you suggest to go about this? Thank you.

abhash-er commented 1 year ago

Would you like to create a benchmark from NASBench201 Search Space's arches or the ZC scores? You can use Trainer in naslib/defaults to run and create benchmarks. If you want to create ZeroCost benchmarks, you can simply use ZeroCost available in predictors.

from naslib.predictors import ZeroCost
pred_type = "jacov"
zc_predictor = ZeroCost(method_type=zc_pred)
score = zc_predictor.query(graph=graph, dataloader=data_loader) 
# The graph is of type NASBench201 type in your case and dataloader would be a subset of CIFAR10 data.

Let me know, if that answers your issue.