NNI v2.8 Iteration Plan

[scarlett2018]

Release Plan for vNext release

• Release manager: @QuanluZhang
• Release start date: tbd
• Feature freeze date: 5.11
• Code freeze and demo date: ~5.18~ 5.20
• Branch cut and next release planning date: tbd
• Bugbash: ~6.1~ ~6.10~ 6.13
• Release date: ~6.13~ ~6.17~ 6.20

Planned Items

NAS

owners: @ultmaster / @QuanluZhang / @JiahangXu / @hzhua

• [x] P0 Merge dev-oneshot back to master. #4760 @ultmaster
• [x] P0 type hint 4.22 #4774 @ultmaster
• [x] P0 [Doc] translation framework document #4759 @ultmaster
• [ ] P0 NAS experiment @QuanluZhang
  • [ ] adopt Zhe's refactor on experiment ~4.29~ ~5.6~ 5.10 #4841
• [x] P0 type hint for one-shot 4.30 #4811 @ultmaster
• [x] P0 Test correctness of final architecture in search space hub. 5.5 @ultmaster #4831
• [x] P1 CGO supports pytorch-lightning 1.6 @hzhua #4814
• [ ] P1 Multi-GPU support for one-shot NAS. #4603 （external contributor: @Frandium)
• [ ] P1 one-shot supports repeat and cell API #4835

Pipeline

• [x] P1 Setup full tests with scale-set agents. @ultmaster
  • [x] placeholder #4846
  • [x] vmss for windows
• [ ] P0 replace docker hub @liuzhe-lz

HPO

owner: @liuzhe-lz

• [x] P1 - Make config.tuner.name case insensitive and let it accept advisors (1 day)

  • 4773

• [x] P1 - Fix TPE import for old metrics

  • 4804

  • 4865

• [x] Fix TPE nested search space

  • 4810

  • 4866

• [x] Fix merge_parameter()

  • 4837

  • 4868

Experiment

owner: @liuzhe-lz

• [x] P0 - Fix Python logging (~1 week, waiting discussing) 4.26

  • 4499

  • [x] #4792
• [x] Refactor NNI manager globals
  • [x] #4703
  • [x] #4705
  • [x] #4706
  • [ ] Check it provides proper message when initializing fails due to remote nni-experiments IO
• [x] Migrate dispatcher IPC to WebSocket (depends on global shutdown)
  • #4660
  • [x] #4806
  • [x] #4808
  • [x] #4821
  • [x] #4825
• [x] Support multiple HPO experiments in one process

  • 4855

• Deferred - Fix experiment management file lock (5 days)
  • [x] #4824
• [x] P1 - Use launcher's sys.executable to detect Python interpreter. Don't blindly use python3.

  • 4799

  • 4840

• Deferred - Support non-English file name

  • 4805

WebUI

owner: @Lijiaoa

• [x] kill job component(code refactor) #4772
  • [x] fix detail page update issue
  • [x] refactor kill job
• [x] P0 log component(code refactor) (Zhe for the help) 4.21 #4794
• [x] P0 support searching customized string in the search box 4.28 #4823
• [x] P1 design discussion, hyper-parameter ordering on webui
• [x] P1 Hyper-parameter overview top trials order

  • 4839

  • 4856

Compression

owners: @J-shang /@zheng-ningxin

• [ ] P0 add a tutorial for config list ~5.10~
• [ ] P0 nni auto compress experiment @J-shang
  • [x] design review 4.20
  • [ ] first version implementation ~4.27~ 4.29 #4836
• [x] P0 type hint 4.26 #4800 @J-shang
• [x] P0 support RecursiveScriptModule in speedup #4801 @J-shang
• [ ] P1 support block sparse in level pruner #4812 @J-shang
• [x] P1 support customization of module replacement function in model speedup (refer to the interface of super-kernel) @zheng-ningxin 5.4 #4826

Deferred

• [ ] P3 - step 6 - global REST handler register (depends on WebSocket and experiment management refactor)
• [ ] P1 pruning config list extension @J-shang
  • [ ] design review ~4.22~
• [ ] P0 Tutorial for one-shot NAS. (depends on 4760) #4509 @ultmaster @JiahangXu ~4.27~ ~5.6~ ??
• [ ] strategies respect resource constraints

[matluster]

NAS Candidates in the scrum

Done and in progress will be move to the root comment

• [ ] P1 Test support of one-shot strategies to search space hub.

Doc

• [ ] P2 tutorials build in pipeline

Pipelines

• [ ] P1 Isolate framework tests from algorithm tests.

[liuzhe-lz]

HPO Candidates in the scrum

Done and in progress will be move to the root comment

• More tutorials (depends on logging)
  • [ ] P1 - Scikit-learn (1.5 days)
  • [ ] P2 - Framework independent (1.5 days)
  • [ ] P2 - Distributed trial @QuanluZhang
• [ ] P1 - Add search space factory API (implementation 1 day, design needs discussion)

Experiment

• [ ] P2 - websocket backend (soft depends on global REST handler)

Document

• [ ] P2 doc reference class doc refactor
• [ ] P1 nnictl overview and fix other comments @QuanluZhang

[Lijiaoa]

WebUI Candidates in the scrum

Done and in progress will be move to the root comment

Feature:

• [ ] support websocket in ui, this task is parallel with Liuzhe's great work websocket
• [ ] experiment comparison UI and backend
  • [x] design: https://www.figma.com/file/T7HTc67K4dmyUBS9cnNpBJ/0427-Full-process?node-id=0%3A1
  • [x] demo from webui
  • [ ] backend support
  • [ ] webui support
• [ ] (discuss)show dict keys in default metric graph and now only support show final result default key (from user)

Refactor and bug fixed:

• [ ] fixing dict intermediate model bug bug report from https://github.com/microsoft/nni/discussions/4289#discussioncomment-2683595
• [ ] refactor compare model refresh status: close refresh when trial status is Done, so trial status RUNNING will continue to refresh the model
• [ ] now React function component use its propTypes to do the type check but use ts interface is more suitable. In summary, use ts interface to describe props to instead of react propTypes check. from liuzhe comments: https://github.com/microsoft/nni/pull/4794#discussion_r861694527
• [ ] add MIT license for webui @liuzhe-lz

[J-shang]

Compression Candidates in the scrum

Done and in progress will be move to the root comment

• [ ] P1 pruning transformer
  • [ ] soft & block movement pruning
  • [ ] merge aligned pruning head and dependence-aware mode to dataflow-aware mode
  • [ ] support special module replacement
• [ ] P1 quantization framework refactor
• [ ] P1 FLOPs counter supports masks
• [ ] P1 test pruning in IT
• [ ] P2 Net Adapt Pruner
• [ ] P2 V2 Model Compression experiment result

[QuanluZhang]

bugbash:

NAS

• [x] P0 Merge dev-oneshot back to master. #4760 @ultmaster tester: @QuanluZhang
• [x] P0 NAS experiment @QuanluZhang tester: @ultmaster
  • [x] adopt Zhe's refactor on experiment ~4.29~ ~5.6~ 5.10 #4841
• [x] P0 Test correctness of final architecture in search space hub. 5.5 @ultmaster #4831 teser: @QuanluZhang
• [x] P1 CGO supports pytorch-lightning 1.6 @hzhua #4814 tester: @QuanluZhang
• [x] P1 Multi-GPU support for one-shot NAS. #4603 （external contributor: @Frandium) tester: @QuanluZhang
• [x] P1 one-shot supports repeat and cell API #4835 tester: @QuanluZhang

HPO

owner: @liuzhe-lz tester: @J-shang

• [x] P1 - Make config.tuner.name case insensitive and let it accept advisors (1 day)

  • 4773

• [x] P1 - Fix TPE import for old metrics

  • 4804

  • 4865

• [x] Fix TPE nested search space

  • 4810

  • 4866

• [x] Fix merge_parameter()

  • 4837

  • 4868

• [ ] error message when wrong field in tuner section. tester: @J-shang

Experiment

owner: @liuzhe-lz

• [x] P0 - Fix Python logging (~1 week, waiting discussing) 4.26 tester: @ultmaster
  • [x] #4499
  • [x] #4792
• [x] Support multiple HPO experiments in one process tester: @J-shang

  • 4855

• [x] P1 - Use launcher's sys.executable to detect Python interpreter. Don't blindly use python3. tester: @yuhuixii

  • 4799

  • 4840

WebUI

owner: @Lijiaoa tester: @liuzhe-lz

• [x] kill job component(code refactor) #4772
  • [x] fix detail page update issue
  • [x] refactor kill job
• [x] P0 log component(code refactor) (Zhe for the help) 4.21 #4794
• [ ] P0 support searching customized string in the search box 4.28 #4823
  • [ ] by default only trial id, parameter id can be searched in search box (next release)
  • [ ] no search button in search box
• [ ] P1 design discussion, hyper-parameter ordering on webui (need linux machine, debug a windows machine)
• [x] P1 Hyper-parameter overview top trials order

  • 4839

  • 4856

Compression

owners: @J-shang /@zheng-ningxin

• [x] P0 nni auto compress experiment @J-shang tester: @liuzhe-lz
  • [x] first version implementation ~4.27~ 4.29 #4836
• [x] P0 support RecursiveScriptModule in speedup #4801 @J-shang tester: @zheng-ningxin
• [x] P1 support customization of module replacement function in model speedup (refer to the interface of super-kernel) @zheng-ningxin 5.4 #4826 tester: @J-shang

More prs:

• [x] test foreground, starting from python api and command line; use abnormal python interpreter to start experiment @liuzhe-lz tester: @yuhuixii
• [x] add replace module in model speedup @yuming tester: @J-shang
• [x] support expand-as @J-shang tester: @zheng-ningxin
• [x] autoformer search space @nbl97 tester: @ultmaster
• [x] supported output_padding for contranspose2d in model speedup @external tester: @J-shang
• [x] move ExperimentManager and TensorboardManager to "extensions" @liuzhe-lz tester: @liuzhe-lz

Pipelines: @ultmaster @J-shang @liuzhe-lz

• [x] windows pipeline @ultmaster
• [x] one bug in training service found by pipeline @ultmaster
• [x] remote hybrid: disk space very limited
• [ ] k8s training services keep as is, fix vm @liuzhe-lz (defer)
• [ ] openpai pipeline @liuzhe-lz (defer)
• [ ] full test, nas and hpo @ultmaster
  • [ ] hpo (issues: has redundancy tests, miss the tests about accuracy) @liuzhe-lz (defer)
  • [ ] move some tests out of ut
  • [x] nas @ultmaster
  • [ ] compression: move tests out of ut @J-shang
  • [ ] write integration tests for compression v2 (defer)
• [x] nas experiment ut @QuanluZhang
• [x] triggered test @J-shang
• [x] daily trigger @ultmaster

[matluster]

Logging:

• All logs appear twice in console.
• Logs on console doesn't have logger name. Expected?

NAS experiment: hang in this state. Experiment status is DONE.

When I press Ctrl-C, the traceback is:

^CException ignored in: <module 'threading' from '/home/azureuser/miniconda3/envs/nni/lib/python3.8/threading.py'>
Traceback (most recent call last):
  File "/home/azureuser/miniconda3/envs/nni/lib/python3.8/threading.py", line 1388, in _shutdown
    lock.acquire()
KeyboardInterrupt: 
[2022-06-01 00:09:19] Stopping experiment, please wait...
INFO:nni.retiarii.experiment.pytorch:Stopping experiment, please wait...
[2022-06-01 00:09:19] Dispatcher exiting...
INFO:nni.runtime.msg_dispatcher_base:Dispatcher exiting...
[2022-06-01 00:09:20] Dispatcher terminiated
INFO:nni.runtime.msg_dispatcher_base:Dispatcher terminiated
[2022-06-01 00:09:20] Experiment stopped
INFO:nni.retiarii.experiment.pytorch:Experiment stopped

[J-shang]

WebUI (from Zhe's comments):

• optimization curve, min/max
• search need an enter button
• download trials info csv, json

Experiment Tuner Config need to report right error message.