NNI v2.9 Iteration Plan

[ultmaster]

• Release manager: @ultmaster
• Release start date: 6.22
• Feature freeze date (at most 30 days): ~7.22~ 8.1
• First test package (1 week since feature freeze): ~7.29~ ~8.12~ 8.16
• Release date (3 weeks since feature freeze): ~8.12~ 9.7

Top selling points

• Official announcement of NAS search space hub
• Support training / search of AutoFormer (BigNAS-styled one-shot NAS)
• Support pruning of transformer: User experience and performance preliminary optimization (masks generation/movement pruning)

Other nominations

• Pythonic HPO search space factory
• Sklearn and Framework-independent tutorial
• Compression supports Lightning: no need to rewrite code when users use lightning in the first place

Other items

NAS

• [ ] P0 - Official announcement of search space hub @ultmaster
  • [x] Setup cluster
  • [x] A table summarizing the performance of popular one-shot strategies ~7.8~ ~7.29~ 8.1 #5034
  • [x] Documentation 8.1 #5035
  • [x] #5051
  • [ ] #5049
  • [x] #5050
  • [ ] #5052
  • [ ] #5053
• [ ] P0 - FLOPs estimator for NAS @ultmaster
  • [x] design review ~7.11~ 7.18
• [x] P0 - Support training (PAO TONG) of AutoFormer (BigNAS-styled one-shot NAS) Renjie ~#4965~ #4987
• [ ] P0 - Support evolution search of AutoFormer ~7.27~ ~8.1~ 8.5 #5054
• [ ] P0 - Promote nni.retiarii to nni.nas @ultmaster ~7.1~ 7.8
  • [x] #4976
  • [x] Move retiarii to NAS #5014
  • [ ] NasExperiment interface (deferred)
• [ ] P1 - Refactor of nni.nas.nn @ultmaster
  • [ ] design review (3rd round)
• [ ] P1 - Tutorial for one-shot NAS. (depends on 4760) #4509 @ultmaster @JiahangXu ~4.27~ ~5.6~ ??
• [ ] P2 - Strategy refactor: strategies respect resource constraints
• [ ] P0 - Refactor experiment @QuanluZhang 7.8
  • [ ] Support view of NAS experiment ~7.1~ #4985
  • [ ] Support resume
  • [ ] Support new interface of experiment

Compression @J-shang

• [x] P0 - Support Block Sparse #4932
• [x] P0 - Evaluator - step 1 : add evaluator #4950
• [x] P0 - Evaluator - step 2 : support evaluator ~7.6~ 7.11 #4992
• [ ] P0 - Movement Pruner Improvement & give a good example (transformer) ~7.20~ ~7.22~ 7.29
• [ ] P1 - Evaluator - step 3 : tutorial and doc ~7.13~ ~7.15~ 7.26 #5016
• [x] P1 - Balance Sparse Improvement #5033
• [ ] ~P1 - Combining all of the above features~
• [ ] P2 - ConfigList Improvement
• [ ] P2 - Wrapper Refactor : support pruning target / exclude refactor / block sparse
• [ ] P2 - Improve Compression Experiment (If have time...)
• [x] P1 - Improve UT & IT

Speedup @louis-j

• [ ] Auto-generate speed up ops
  • [ ] First draft ~7.11~ 7.13 #4996
  • [ ] Test different layers and torch versions

Quantization

• [ ] Quantization refactor @QuanluZhang ~7.22~ ~7.29~ 8.1

HPO @liuzhe-lz

• [ ] P1 - Scikit-learn tutorial (1 day) ~6.27~ ~7.4~
• [ ] P1 - Framework independent tutorial (1 day)
• [ ] P1 - Pythonic search space API (2 days)

Experiment @liuzhe-lz

• [ ] P0 - websocket reconnect
• [ ] P2 - logging refactor
  • [ ] dispatcher log (1 day)
  • [ ] trial log (2 days)
• [ ] P3 - Split NNI manager client APIs and experiment management APIs (2 days)

WebUI

P0 - 15 days

• [x] P0 - #4973 6.30
• [x] #4975
• [x] P0 - Bug fix: after clicking compare button, the dropdown button (for selecting different keys) is not shown. 7.8 #4990
• [x] Fix issue #4969 ~7.13~ 7.15 #5011
• [x] P0 - Refactor: make React function component use TS interface, use TS interface to describe props. 7.27 #5029
• [x] P0 - Show error/warning messages at the bottom right of the page with small popup windows. 7.28 #5029
• [ ] Discuss: search box
• [ ] Design: the location of trial log URL/button
• [ ] Manage/control experiments on WebUI (maybe create experiments)
• [ ] Project view on WebUI
• [ ] Reduce WebUI latency / Improve trial status consistency
  • [ ] WebUI based on WebSocket

Training service

• Training service refactor 7.25
  • [ ] Local training service on PoC entrance 7.10 (review meeting 7.11)
  • [ ] Compatibility layer between new training service and old infrastructure
  • [ ] New remote training service
  • [ ] Remote reconnect #514
  • [ ] New infrastructure
  • [ ] Compatibility layer between new infrastructure and old training services
  • [ ] (stretch) Raw k8s training service
• P0 - Support custom docker registry
  • [ ] Kubeflow
  • [ ] FrameworkController
  • [ ] OpenPAI
• [ ] Support of Microsoft-internal training service

Pipelines

• [x] P0 - 1ES related stuffs
  • [x] Create image and pool @QuanluZhang @ultmaster
  • [x] Use Microsoft-hosted pool for full tests, temporarily @J-shang
  • [x] apt-get locked by unattended-upgrade
  • [x] NAS experiment hang on Windows for unknown reason
  • [x] GPU sometimes offline after several minutes (local linux)
  • [x] Need a new compliant image (new resource group required)
  • [x] Windows doesn't have GPU driver
• [ ] Move tuner test to full test HPO
• [ ] K8S pipeline @liuzhe-lz @QuanluZhang

Deferred from last release

• [ ] P3 - step 6 - global REST handler register (depends on WebSocket and experiment management refactor)
• [ ] P1 pruning config list extension @J-shang
  • [ ] design review ~4.22~

[Simba98]

Notice that the training service is being refactored. Should we consider fix #514?

[J-shang]

• [ ] test lr_scheduler, max_steps, max_epochs work well in Evaluator.train(...). #4950
• [ ] test all pruner old API (without Evaluator) (deprecation warning) #4992
• [ ] test TorchEvaluator and LightningEvaluator work well in all pruners #4950 #4992 #5016
• [ ] test balance mode in LevelPruner #5033
• [ ] test dependence aware mode only generate the same channel-wise masks if two modules have dependence in related pruners #4992
• [ ] test global mode with key max_sparsity_per_layer can indeed limit the maximum sparsity in related pruners #5017
• [ ] test MovementPruner new API training_epochs training_steps regular_scale movement_mode sparse_granularity #5017
• [ ] test the transformer pruning example can achieve the reported number. #5017
• [ ] test auto convert op ast to callable #4996
• [ ] bug fix (replace Linear, LayerNorm in both [batch, hidden] and [batch, seq, hidden] senior) #4925 #5017

known issue: activation based pruner can not support [batch, seq, hidden]

[ultmaster]

v2.9 Release

Neural Architecture Search

Test items

• [x] Run DARTS and NAS-Bench-201 (multiple algo) example in https://github.com/microsoft/nni/pull/5052 @QuanluZhang
• [x] View and resume of NAS experiment. @ultmaster
• [x] Autoformer @ultmaster
• [x] Tutorial of DARTS @JiahangXu

Highlights

• Official release of model space hub.
• Tutorial of searching in DARTS.
• Support view / resume of NAS experiment.
• Autoformer (not ready yet).

Features

• Support fit_kwargs in lightning evaluator.
• Support drop_path and auxiliary_loss in NASNet.
• Add export_probs to monitor the architecture weights.
• Rewrite configure_optimizers, functions to step optimizers / schedulers, along with other hooks for simplicity, and to be compatible with latest lightning (v1.7).
• Support gradient clipping in DARTS.
• Align implementation of DifferentiableCell with DARTS official repo.
• Re-implementation of ProxylessNAS.

Bug fixes

• Fix a misuse of lambda expression in NAS-Bench-201 search space.
• Fix the gumbel temperature schedule in Gumbel DARTS.
• Fix a performance issue caused by tensor formatting in weighted_sum.
• Fix the architecture weight sharing when sharing labels in differentiable strategies.
• Fix the memo reusing in exporting differentiable cell.

Compression

• [x] test lr_scheduler, max_steps, max_epochs work well in Evaluator.train(...). #4950 @QuanluZhang
• [x] test all pruner old API (without Evaluator) (deprecation warning) #4992 @QuanluZhang
• [x] test TorchEvaluator and LightningEvaluator work well in all pruners #4950 #4992 #5016 @QuanluZhang
• [x] test TransformersEvaluator @QuanluZhang
• [x] test GroupNorm speedup @J-shang
• [x] test balance mode in LevelPruner #5033 @zheng-ningxin
• [x] test dependence aware mode only generate the same channel-wise masks if two modules have dependence in related pruners #4992 @Louis-J
• [x] test global mode with key max_sparsity_per_layer can indeed limit the maximum sparsity in related pruners #5017 @Louis-J
• [x] test MovementPruner new API training_epochs training_steps regular_scale movement_mode sparse_granularity #5017 @Louis-J
• [x] test the transformer pruning example can achieve the reported number. #5017 Xin Zhang
• [x] test auto convert op ast to callable #4996 @J-shang
• [x] bug fix (replace Linear, LayerNorm in both [batch, hidden] and [batch, seq, hidden] senior) #4925 #5017 @J-shang

known issue: activation based pruner can not support [batch, seq, hidden] replace doc-compression-overview-Compression Pipeline-picture

HPO

• [x] Fix the bug that weights are not defined correctly in adaptive_parzen_normal of TPE. @liuzhe-lz

Training service

• [x] Frameworkcontroller + NFS

• [ ] Need more comprehensive tests (either pipeline or manual combinations)

• [x] Fix trialConcurrency bug, use ${envId}_run.sh to replace run.sh

• Fix upload dir bug in K8S training service: use a separate working directory for each experiment. @liuzhe-lz

• Fix trialConcurrency bug, use ${envId}_run.sh to replace run.sh. Thanks external contributor @liuzhe-lz

Web portal @ultmaster

• [x] Fix the issue of None / null display in tensorflow example.
• [x] Fix an issue in experiment error status
• [x] Upgrade react-router to v6: test react-router index router with normal local experiment and with url_prefix experiment
• [x] Test dispatch log/nnimanager log page with normal local experiment and with url_prefix experiment
• [x] Test trial compare model and trial intermediate result chart is right with normal results and dict (intermediate/final) results
• [x] Test details page
• [x] Test overview page

Internal refactors and improvements

• Move nni.retiarii code-base to nni.nas.

[ultmaster]

Release note

Neural Architecture Search

• New tutorial of model space hub and one-shot strategy (tutorial)
• Add pretrained checkpoints to AutoFormer (doc)
• Support exporting state dict of subnet from supernet (doc)
• Support view and resume of NAS experiment (doc)

Enhancements

• Support fit_kwargs in lightning evaluator (doc)
• Support drop_path and auxiliary_loss in NASNet (doc)
• Support gradient clipping in DARTS. (doc)
• Add export_probs to monitor the architecture weights.
• Rewrite configure_optimizers, functions to step optimizers / schedulers, along with other hooks for simplicity, and to be compatible with latest lightning (v1.7).
• Align implementation of DifferentiableCell with DARTS official repo.
• Re-implementation of ProxylessNAS.
• Move nni.retiarii code-base to nni.nas.

Bug fixes

• Fix a performance issue caused by tensor formatting in weighted_sum.
• Fix a misuse of lambda expression in NAS-Bench-201 search space.
• Fix the gumbel temperature schedule in Gumbel DARTS.
• Fix the architecture weight sharing when sharing labels in differentiable strategies.
• Fix the memo reusing in exporting differentiable cell.

Compression

• New tutorial of pruning transformer model (tutorial)
• Add TorchEvaluator, LightningEvaluator, TransformersEvaluator for easier passing training logic to pruner (doc, API)

Enhancements

• Promote all pruner API using Evaluator, the old API is still remained and will be deprecated in v3.0 (doc)
• Support most of ops in pruning speedup by automatic op conversion.
• Support lr_scheduler in pruning by using Evaluator.
• Support pruning NLP task in ActivationAPoZRankPruner and ActivationMeanRankPruner.
• Add training_steps, regular_scale, movement_mode, sparse_granularity for MovementPruner (doc)
• Add GroupNorm replacement in pruning speedup. Thanks external contributor @cin-xing
• Optimize balance mode performance in LevelPruner.

Bug fix

• Fix dependency_aware mode invalid in scheduled pruners.
• Fix bias mask cannot be generated.
• Fix max_sparsity_per_layer not take effect.
• Fix Linear and LayerNorm speedup replacement in NLP task.
• Fix tracing LightningModule failed in pytorch_lightning >= 1.7.0

Hyper-parameter optimization

• Fix the bug that weights are not defined correctly in adaptive_parzen_normal of TPE.

Training service

• Fix trialConcurrency bug in K8S training service: use ${envId}_run.sh to replace run.sh
• Fix upload dir bug in K8S training service: use a separate working directory for each experiment. Thanks external contributor @amznero

Web portal

• Support dict keys in Default metric chart in the detail page
• Show experiment error message with small popup windows in the bottom right of the page
• Upgrade React router to v6 to fix index router issue
• Fix the issue of details page crashing due to choices containing None
• Fix the issue of missing dict intermediate dropdown in comparing trials dialog

Known issues

• Failed trials are NOT auto-submitted when experiment is resumed (#4931 is reverted due to its pitfalls).
• Speedup rand_like_with_shape is easy to overflow when dtype=torch.int8.
• Speedup doesn't support registered buffer.
• Speedup doesn't support list/dict input.