ProxylessNAS example accuracy and loss not updated + strategy.Proxyless() support

[mahdihey]

Describe the issue: I'm running running ProxylessNAS example. The accuracy and loss haven't been updated after 112 epochs. What is the expected behavior?

I found out that in 2.8 version strategy.Proxyless() was added in place of deprecated ProxylessTrainer(). Is there a plan to create a similar ProxylessNAS example? Or, how I can modify it for strategy.Proxyless()?

Environment:

• NNI version: 2.8
• Training service (local|remote|pai|aml|etc): local
• Client OS: Linux
• Server OS (for remote mode only):
• Python version: 3.6
• PyTorch/TensorFlow version: 1.7
• Is conda/virtualenv/venv used?: yes
• Is running in Docker?: no

Configuration:

• Experiment config (remember to remove secrets!):
• Search space:

Log message:

• nnimanager.log:
• dispatcher.log:
• nnictl stdout and stderr:

How to reproduce it?:

[ultmaster]

What is your search space? If it only contains LayerChoice and InputChoice, you can directly use the latest strategy.Proxyless().

Please note that the ProxylessNAS before v2.8 is known to be buggy. We rewrite the Proxyless in v2.9. You can checkout the master branch if you want to have a try.

[mahdihey]

I'm trying to perform exactly https://github.com/microsoft/nni/tree/master/examples/nas/oneshot/proxylessnas example. Using the same search space on mobilenet-like model and imagenet dataset. Is it possible to do the same exercise on the strategy.Proxyless()? If so, what changes should be done in the proxylessnas example?

[mahdihey]

How is the stability and robustness of strategy.Proxyless()? Has it been tested against some reference?

[matluster]

I'm trying to perform exactly https://github.com/microsoft/nni/tree/master/examples/nas/oneshot/proxylessnas example. Using the same search space on mobilenet-like model and imagenet dataset. Is it possible to do the same exercise on the strategy.Proxyless()? If so, what changes should be done in the proxylessnas example?

I think you might want to try strategy.Proxyless() on nni.retiarii.hub.pytorch.ProxylessNAS() space. But I think it won't work because: (1) Repeat is not supported by Proxyless strategy; (2) Latency loss is not supported. I have to admit this is kind of regression as of v2.8, but as I said in #5056, we have a plan for systematic support coming soon. Plus, the original version is known to be troublesome for a long time. Honestly I didn't anticipate many active users...

How is the stability and robustness of strategy.Proxyless()? Has it been tested against some reference?

I think I've tested ProxylessNAS on NAS-Bench-201 in #5052. It works fine in initial iterations, but I forgot to collect the final results (busy around other things recently). Though NB-201 is a toy search space, but believe it or not, the original ProxylessTrainer doesn't work on NB-201 at all. :(

If you are interested in, for example, supporting Repeat API for Proxyless strategy, contributions are always welcomed.