Group 4: Porting Zero-cost NAS proxies to MASE

Porting Zero-cost NAS proxies to MASE

We added a proxy action and proxy_strategy in search action, which allows integration of zero-cost proxies with existing bayesian based search algorithms.

Overview

Proxy

Added proxy action in MASE command line interface, major changes in path machop/chop/actions/proxy/...
Integrated NasBench201 and NasBench301 search spaces into MASE proxy action
Integrated 13 zero cost proxies, including fisher, jacov, params, etc... from ℕ𝔸𝕊-𝔹𝕖𝕟𝕔𝕙-𝕊𝕦𝕚𝕥𝕖-ℤ𝕖r𝕠 into MASE proxy action
Included sample configuration for proxy action in path machop/configs/nas/proxy_nas.toml
Included demo for training a meta-proxy from NasBench201 search space on CIFAR10 dataset

Search with proxy

Added proxy search strategy in MASE search action on Val Accuracy metric, major changes in path machop/chop/actions/search/strategies/strats_proxy.py
Integrated 13 zero cost proxies, including fisher, jacov, params, etc... from ℕ𝔸𝕊-𝔹𝕖𝕟𝕔𝕙-𝕊𝕦𝕚𝕥𝕖-ℤ𝕖r𝕠 into MASE search action, proxy_strategy search strategy
Included sample configuration for proxy search strategy in path machop/configs/nas/search_nas.toml

How to use

Proxy Action Workflow

When proxy action is ran in MASE cli, architectures will be randomly sampled from NasBench201 or NasBench301 (your choice) search spaces.
Zero-cost proxy scores will be calculated for each architectures using proxies from ℕ𝔸𝕊-𝔹𝕖𝕟𝕔𝕙-𝕊𝕦𝕚𝕥𝕖-ℤ𝕖r𝕠
Using validation accuracy values from NasBench, a model will be trained to predict validation accuracy from zero-cost proxy scores
The model, which is the MetaProxy, will be saved as a .pt file

Sample `proxy` action

Set directory:

cd machop

Use the following command to execute proxy action with settings from our sample configuation file.

./ch proxy --config configs/nas/proxy_nas.toml

The resultant weights are saved in path nas_results/meta_proxy/meta_proxy.pt

Search with proxy

In the search space you desire to search, MASE will first build a graph for each architecture
Zero-cost proxy scores will be calculated for each graph using proxies from ℕ𝔸𝕊-𝔹𝕖𝕟𝕔𝕙-𝕊𝕦𝕚𝕥𝕖-ℤ𝕖r𝕠
The scores will then be fed to the Meta Proxy creaded from proxy action, to estimate accuracy values for each architecture
Baysian search will be conducted based on the estimated accuracy

Sample `search` action with `proxy` strategy

Set directory:

cd machop

Then, using the following command, search with zero-cost proxy could be executed with our sample configuation file.

./ch search --config configs/nas/search_nas.toml

Expected output:

Best trial(s):
|    |   number | software_metrics            | hardware_metrics                                | scaled_metrics              |
|----+----------+-----------------------------+-------------------------------------------------+-----------------------------|
|  0 |        0 | {'accuracy': '[0.8809148]'} | {'average_bitwidth': 32, 'memory_density': 1.0} | {'accuracy': '[0.8809148]'} |

Others

Fixed bug on invalid search iteration
Included proxy scores for 8901 architectures sampled from NasBenchNLP
Added sphinx docs documentation in path machop/sphinx_docs/source/modules/documentation/tutorials/actions/proxy and machop/sphinx_docs/source/modules/documentation/tutorials/actions/search/how_to_extend_search.md

Future work

Extend proxy search strategy on other metrics, such as validation loss values of models
Include more search spaces or user defined search spaces for proxy action

DeepWok / mase