search

automl / NASLib

NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.
Apache License 2.0
512 stars 117 forks source link

Zero-Cost Predictor - Run on all 15,625 architecture for NasBench201 #142

Closed alec-flowers closed 1 year ago

alec-flowers commented 1 year ago

Hi, I wanted to test a Zero-Cost Predictor on all 15,625 architectures for NASBench201. In the NAS-Bench-Suite-Zero paper you state that you did this for all the architectures -

For all NAS-Bench-201 and TransNAS-Bench-101 tasks, we evaluate all ZC proxy values and the respective runtimes, for all architectures

But I am running across some issues. In the predictor_config.yaml it only has an option for random sampling of architectures with the parameter uniform_random.

I wrote my own code to exhaustively sample all architectures using _naslib.search_spaces.nasbench201.graph.get_architerator(). However, _PredictorEvaluator.loaddataset() then loads 15,625 graphs into a list, which around 6,000 architectures in, requires too much memory for my computer. I wanted to see if I was going about this the best way or if there was a better way.

Thanks.

Neonkraft commented 1 year ago

Hi @alec-flowers

Thank you for your interest in our project! You'll find that all the code relating to zero cost proxies are in the zerocost branch. You can edit this runner file to run your proxies on all the nb201 models.

Best, Arjun