NNI v3.0 preview iteration plan

[scarlett2018]

Release manager: @liuzhe-lz Release start date: 10.31 Feature freeze date (at most 30 days): 2.10 2.20 Code freeze date & first package: 2.17 2.24 2.28 Tutorial freeze: 2.24 3.3 Release date (3 weeks since feature freeze): 3.3 3.10

Misc

• NAS Fine-tune
• HPO integrated to other tool
• Speed up support with Graph (Chunyang)

[liuzhe-lz]

NAS

• [x] P1.0 - Mutable search space (coding done) 11.4

  • 5194

  • 5195

  • 5196

  • 5197

  • 5198

  • 5199

  • 5247

  • 5200

• [ ] Space format

  • 5267

  • 5268

  • 5269

  • 5270

• [ ] P1.1 - Refactor nas.nn (1 week)

  • 5279

  • 5280

  • 5281

  • 5282

• [ ] Trial command channel

  • 5254

• [ ] Refactor hardware-ware NAS
  • [ ] P1.4 - Refactor (waiting PR)
  • [ ] One more example (TBD) ?
• [ ] Execution & experiment
• [ ] Evaluation of existing strategies & models (need intern)

[liuzhe-lz]

Compression

Something

• [ ] P0 - Wrapper refactor (2 weeks) 11.25 12.2 12.7

  • 5151

• [ ] P2 - Config key redesign
• [ ] P0 - Framework
• [ ] P1 - Support algorithms (half week)
• [ ] P1 - System support block-sparse (2 days)
• [ ] P2 - Support more module type in compression

Common

• [x] P0 - Remove pruning v1 & refactor compression directory (2 days)

  • 5228

• [ ] P1 - Fully test & support for DDP in compression @Bonytu
  • [x] Fix hook-based pruner multi-device data out of sync 11.24

  • 5242

  • [ ] DDP Example 11.23
• [ ] P2 - Integrate with fvcore/thop

Disillation

• [x] P0 - Offline Util TBD@11.10

  • 5215

Pruning

• [ ] P1 - Huggingface extension (-)

  • 5137

Pruning speed up

• [x] P0 - torch.fx graph generation (Chunyang) 11.4 11.30

  • 5121

• [ ] graph generation part 2
• [ ] P0? - (design) modelspeedup v2

  • 5250

• [ ] P0 - Transformer speedup
  • [x] code

  • 5238

  • [ ] doc

    Quantization

• [ ] P2 - Refactor

  Quantization speed up

• [ ] P1 - Improve QS on tensorRT (2 weeks) 11.11 11.18

  • 5039

• [ ] P1 - Support QS on ORT
  • [ ] survey (1 week)
• [ ] P2 - New PTQ algo AdaRound

[liuzhe-lz]

Fix Bug

• [ ] Experiment resume @QuanluZhang

HPO

• [ ] @ultmaster
  • [ ] Refactor platform

  • 5243

  • [x] Remove HPO annotation

  • 5249

• Fix nnictl update xxx @liuzhe-lz
• Change default optimize_mode (1 day)
• Pythonic search space factory
• Sklearn tutorial

NNI Manager

• [x] Framework
  • [x] Update node.js

  • 5206

  • [x] v3 as v1 compat layer

  • 5211

  • [ ] v1/v2 as v3 compat layer?
• [ ] Utils
  • [x] HTTP command channel

  • 5216

  • [x] WebSocket command channel 12.20
  • [x] GPU scheduler library

  • 5219

  • [ ] GPU scheduler daemon 12.20
  • [x] Trial keeper library 11.14

  • 5229

  • [x] Trial keeper daemon 12.20
• [ ] Local
  • [x] Usable 11.16 11.24

  • 5243

  • [ ] Refined
• [ ] Remote
  • [x] Usable 11.21 12.22
  • [ ] Refined
• [ ] Update NNI manager
  • [ ] Callback based shutdown 12.28
  • [ ] Fix experiment life cycle tests ?
• [ ] Other training services
  • [ ] AML 1.6
  • [ ] Kubeflow 1.13
  • [ ] FrameworkController @QuanluZhang 1.11
  • [ ] amlt @ultmaster
• [ ] Misc
  • [ ] Distributed trial
  • [ ] Log collection
  • [ ] Metric collect (check TensorBoard support)
  • [ ] Shared storage

[liuzhe-lz]

Web UI

• [x] Unify max/min buttons (1 week) 11.8 11.10

  • 5222

• [ ] New design for overview page (1 week) 11.19
  • [x] main

  • 5240

  • [x] nav bar 12.2

  • 5260

• [x] Fix some bugs

  • 5246

• [ ] (2 weeks total)
  • [ ] Show cluster info
  • [ ] Select all & select none
  • [ ] Export trials to CSV
  • [ ] Hide trials from chart

[liuzhe-lz]

Bug bash

NNI manager and HPO experiment

• [ ] Local
  • [ ] Basic
  • [ ] Resume
  • [ ] Log
• [ ] Remote @ultmaster
  • [ ] Basic
  • [ ] Resume (TODO: Should resume killed trials)
  • [ ] Log
  • [ ] Failure recovery (Kill nni processes on worker. It should auto recover)
• [x] Web UI @super-dainiu (https://nni.readthedocs.io/en/latest/tutorials/hpo_quickstart_pytorch/main.html) run experiment with 100 trials:
  • [ ] use maximize/minmize button in all component(top trials, default metric, hyper-parameter)
  • [x] use all things in the overview page, edit experiment-params
  • [x] use nav
  • [x] use detail page 
  • [ ] stop and resume experiment
  • [x] kill trial 'RUNNING', 'ADD_RESUMED' 
  • [x] webui doc* [ ]

Known bugs to be fixed:

• [ ] Web UI trial log path does not work (though viewing log should work)
• [ ] Resuming trials does not work (resuming experiment should work)

NAS

Example: https://github.com/ultmaster/nni/tree/nas-nn-refactor/examples/nas/hub Define a custom model space (prefer not CNN), randomly choose some strategies.

• [ ] Multi-trial strategies @liuzhe-lz
  • [ ] Random
  • [ ] Evolution
  • [ ] RL
• [ ] One-shot @QuanluZhang
  • [ ] 5 algos: https://nni.readthedocs.io/en/latest/nas/exploration_strategy.html
• [ ] Engine: Sequential engine & Training service engine (need to test resume) @juntao
• [ ] Hardware aware (example: https://github.com/ultmaster/nni/blob/nas-nn-refactor/examples/nas/hub/proxyless_search.py#L396 ) @juntao

Compression

• [ ] Pruner @Bonytu
  • [x] first round
  • [ ] second round
• [ ] Quantizer @J-shang
  • [x] first round
  • [ ] second round
  • [ ] fix quant scheme support issue
• [ ] Distiller @hongyi
• [ ] Speed up v2 @super-dainiu
• [ ] Distillation util + doc @penghouwen
• [ ] Fusion compression @QuanluZhang @J-shang (https://github.com/microsoft/nni/blob/master/nni/contrib/compression/base/compressor.py#L62)

[Lijiaoa]

Web UI

• [x] Unify max/min buttons (1 week) ~11.8~ 11.10

  • use global optimized_mode in related components #5222

• [ ] New design for overview page (1 week) 11.19

  • [x] main

  • PR for overview page new design #5240

  • [x] nav bar 12.2

  • add nav options changes to overview page #5260

• [x] Fix some bugs

  • [dev-webui] fix bugs #5246

• [ ] (2 weeks total)

  • [ ] Show cluster info
  • [ ] Select all & select none
  • [ ] Export trials to CSV
  • [ ] Hide trials from chart

+1 update

• [x] select all & select none https://github.com/microsoft/nni/pull/5297

[Lijiaoa]

+1 update

Known bugs to be fixed:

• [x] Web UI trial log path does not work (though viewing log should work)
• [x] Resuming trials does not work (resuming experiment should work)
  • [ ] in resume mode, trial‘s duration will be recalculated (won't fix, not worthy)

[liuzhe-lz]

Release Note

NAS

@ultmaster

Compression

@J-shang

Training Services

• Breaking change: NNI v3.0 cannot resume experiments created by NNI v2.x
• Local training service:
  • Reduced latency of creating trials
  • Fixed "GPU metric not found"
  • Fixed bugs about resuming trials
• Remote training service:
  • reuse_mode now defaults to False; setting it to True will fallback to v2.x remote training service
  • Reduced latency of creating trials
  • Fixed "GPU metric not found"
  • Fixed bugs about resuming trials
  • Supported viewing trial logs on the web portal
  • Supported automatic recover after temporary server failure (network fluctuation, out of memory, etc)

Web Portal

• New look and feel

[Lijiaoa]

V3.0 Related issue

https://github.com/microsoft/nni/issues/5531 √ https://github.com/microsoft/nni/issues/5477 https://github.com/microsoft/nni/issues/5464 https://github.com/microsoft/nni/issues/5452 https://github.com/microsoft/nni/issues/5418 https://github.com/microsoft/nni/issues/5263 √ https://github.com/microsoft/nni/issues/5253 √ https://github.com/microsoft/nni/issues/5239 https://github.com/microsoft/nni/issues/5192 √ https://github.com/microsoft/nni/issues/5046 https://github.com/microsoft/nni/issues/5097 √ https://github.com/microsoft/nni/issues/5090 √ https://github.com/microsoft/nni/issues/5202 √ https://github.com/microsoft/nni/issues/5104 √ https://github.com/microsoft/nni/issues/5064 √

---

https://github.com/microsoft/nni/issues/5143 try v3.0

[matluster]

NAS

• Breaking change: nni.retiarii is no longer maintained and tested. Please migrate to nni.nas.
  • Inherit nni.nas.nn.pytorch.ModelSpace, rather than use @model_wrapper.
  • Use nni.choice, rather than nni.nas.nn.pytorch.ValueChoice.
  • Use nni.nas.experiment.NasExperiment and NasExperimentConfig, rather than RetiariiExperiment.
  • Use nni.nas.model_context, rather than nni.nas.fixed_arch.
  • Please refer to quickstart for more changes.
• A refreshed experience to construct model space.
  • Enhanced debuggability via freeze() and simplify() APIs.
  • Enhanced expressiveness with nni.choice, nni.uniform, nni.normal and etc.
  • Enhanced experience of customization with MutableModule, ModelSpace and ParamterizedModule.
  • Search space with constraints is now supported.
• Improved robustness and stability of strategies.
  • Supported search space types are now enriched for PolicyBaseRL, ENAS and Proxyless.
  • Each step of one-shot strategies can be executed alone: model mutation, evaluator mutation and training.
  • Most multi-trial strategies now supports specifying seed for reproducibility.
  • Performance of strategies have been verified on a set of benchmarks.
• Strategy/engine middleware.
  • Filtering, replicating, deduplicating or retrying models submitted by any strategy.
  • Merging or transforming models before executing (e.g., CGO).
  • Arbitrarily-long chains of middlewares.
• New execution engine.
  • Improved debuggability via SequentialExecutionEngine: trials can run in a single process and breakpoints are effective.
  • The old execution engine is now decomposed into execution engine and model format.
  • Enhanced extensibility of execution engines.
• NAS profiler and hardware-aware NAS.
  • New profilers profile a model space, and quickly compute a profiling result for a sampled architecture or a distribution of architectures (FlopsProfiler, NumParamsProfiler and NnMeterProfiler are officially supported).
  • Assemble profiler with arbitrary strategies, including both multi-trial and one-shot.
  • Profiler are extensible. Strategies can be assembled with arbitrary customized profilers.

[J-shang]

Compression

• Compression framework is refactored, new framework import path is nni.contrib.compression.
  • Configure keys are refactored, support more detailed compression configurations.
  • Support multi compression methods fusion.
  • Support distillation as a basic compression component.
  • Support more compression targets, like input, ouptut and any registered paramters.
  • Support compressing any module type by customizing module settings.
• Pruning
  • Pruner interfaces have fine-tuned for easy to use.
  • Support configuring granularity in pruners.
  • Support different mask ways, multiply zero or add a large negative value.
  • Support manully setting dependency group and global group.
  • A new powerful pruning speedup is released, applicability and robustness have been greatly improved.
  • The end to end transformer compression tutorial has been updated, achieved more extreme compression performance.
• Quantization
  • Support using Evaluator to handle training/inferencing.
  • Support more module fusion combinations.
  • Support configuring granularity in quantizers.
• Distillation
  • [DynamicLayerwiseDistiller]() and [Adaptive1dLayerwiseDistiller]() are supported.
• Compression documents now updated for the new framework, the old version please view v2.10 doc.
• New compression examples are under [nni/examples/compression]()
  • Create a evaluator: nni/examples/compression/evaluator
  • Pruning a model: nni/examples/compression/pruning
  • Quantize a model: nni/examples/compression/quantization
  • Fusion compression: nni/examples/compression/fusion

[ShrutiSarikaChakraborty]

Hi,

The instruction was - "Use nni.nas.model_context, rather than nni.nas.fixed_arch".

However I am getting error for 2.10 version -,

import nni.nas.model_context Traceback (most recent call last):

File "C:\Users\ADMIN\AppData\Local\Temp\ipykernel_14148\1503585595.py", line 1, in import nni.nas.model_context

ModuleNotFoundError: No module named 'nni.nas.model_context'

I checked that nni.nas has NOT got any module named model_context in either v2.10 or v3.0. Please help.

Thanks

[ultmaster]

Hi,

The instruction was - "Use nni.nas.model_context, rather than nni.nas.fixed_arch".

However I am getting error for 2.10 version -,

import nni.nas.model_context

Traceback (most recent call last):

File "C:\Users\ADMIN\AppData\Local\Temp\ipykernel_14148\1503585595.py", line 1, in

import nni.nas.model_context

ModuleNotFoundError: No module named 'nni.nas.model_context'

I checked that nni.nas has NOT got any module named model_context in either v2.10 or v3.0. Please help.

Thanks

Dup of #5585