NNI 2021 August~September Iteration Planning

[QuanluZhang]

This is the plan for the iteration in August~September Iteration, it's a 6 weeks iteration.

Release Plan for stableness release and use case support

• Release manager: @acured
• Feature freeze date: ~Sept 9~ Sep 16
• Code freeze and demo date: ~Sept 16~ ~Sep 23~ ~Sep 28~ Oct 9
• Branch cut and next release planning date: ~Sept 23~ ~Sep 30~ Oct 13
• Bugbash - Oct 22
• Release date: ~Sept 27~ ~Oct 8~ ~Oct 9~ ~Oct 18~ Oct 25

NAS

• [ ] (need review) fix benchmark @ultmaster
  • [x] unify architecture format. Aug 20 #4090
  • [x] provide simple API for query. Aug 20 #4090
  • [ ] support benchmark end2end (test strategies' performance) #4205
• [ ] support serialization @ultmaster #4172
• [x] review and improve nas tests stage 1 @ultmaster. ~Aug 20~ Aug 31 #4097
• [x] improve quick start (notebook) @QuanluZhang. ~Aug 12~ ~Aug 17~ Aug 20 PR #4094
  • [x] replace example, update mnist example. ~Aug 27~ Sep 1
• [ ] examples for image classification, object detection, audio (show reasonably good performance)
  • [ ] transformer @skeletondyh ~Aug 31~
  • [x] investigate search space and common models and chose one to start.
  • [x] review meeting for investigate. Sep 1
  • [x] implement NAS. Sep 8
  • [x] review meeting for result(demo). ~Sep 10~ ~Sep 15~ ~Sep 17~ Sep18
  • [ ] improvement and PR(code and doc) Sep 27 #4222
• [x] deal with pytorch lightning onnx error for NAS visualization @ultmaster

Model Compression

• [ ] support compression v2 @J-shang Aug 20 #3917 #4074 #4089 #4116 #4178
  • [ ] tests #4125
  • [ ] (bug bash)doc tutorial
• [ ] support per-channel quantization #4137
• [ ] end2end examples for image classification, object detection, audio, transformer (show reasonably good performance)
  • [x] image classification (MobileNet) @xiaowu0162 ~Aug 18~ Aug 30 #4102
  • [x] investigate model compression paper and try to compress model by NNI.
  • [x] review meeting for investigate and NNI result(demo).
  • [x] compare with other opensource compression tools.
  • [x] review meeting for comparing results.
  • [x] improvement and PR(code and doc) #4102
  • [ ] object detection (coco, yolo3) @linbinskn ~Sep 1~ Sep 22
  • [x] investigate model compression paper and try to compress model by NNI.
  • [x] review meeting for investigate and NNI result(demo). ~Sep 2~ Sep 8
  • [x] support multi-gpu training @chenbohua3 #4127
  • [x] support bn-folding for lsq quantizer Sep 8 #4148
  • [ ] review meeting for comparing results. Sep 26
  • [ ] improvement and PR(code and doc)

HPO

• [x] refactor Hyper-opt based algorithm - random tuner. @liuzhe-lz #4118

nnictl & nnimanager & training service @SparkSnail @liuzhe-lz @acured

• [ ] P0 - fix macos ut @liuzhe-lz ~Aug 20~ Aug 24 #4109
  • [ ] (bug bash)ut stage 2, modify dependency injection
• [x] P0 - add trial gpu test in UT, verify whether gpu is used in IT @acured #4162
• [x] P0 - hybrid local & remote & aml integration test @SparkSnail Aug 23 #4124
• [x] P0 - clean deprecated code (nasui) @ultmaster (nasui)#4088
• [x] P0 - demo and improve distributed trials on different training services (e.g., kubeflow) @SparkSnail Sep 8
  • [x] Is support? How to support? investigate
• [x] fix pipeline manual az login issue @acured Aug 20 #4082
• [x] fix model speedup out of memory? issue @J-shang
• [x] experiment config doc update @J-shang #4131
• [x] improve doc: clarify the difference between training service storage and shared storage @J-shang #4217
• [x] double check the cancel logic of aml (&pai?) trial runner @SparkSnail

WebUI

• [x] bug fix - Improve trial intermediate result graph. @Lijiaoa #4171
• [x] bug fix - webui prompts error message. @Lijiaoa #4164

regression test needed, or bugs reported during the iteration by external users (need test and close the issues)

• [x] #4077, #4083 Restful server started failed @acured
• [ ] nni==2.3 tensorboard uninstall then install get version erro,how solve it? #4087 @SparkSnail
• [ ] #4079 @Lijiaoa
• [ ] Exponential Notation is not allowed anymore #4080 @QuanluZhang
• [ ] SyntaxError on WebUI with old version of Firefox #4081 @Lijiaoa
• [x] #4158 @acured
• [ ] assert len(set(num_channels_list)) == 1 #4160 @J-shang
• [ ] #4066 users found an walkaround, we need a fix for it @liuzhe-lz
• [ ] Task is always in a waiting state in the remote machine #3905 (not an 2.4 regression but a 2.1 regression) @acured
• [ ] NNI installation issue with ConfigSpace-0.4.7 #3909 (not sure whether this is a regression or not) @liuzhe-lz
• [ ] Runtime Error during L1FilterPruner (and other one-shot pruners) #3944 @zheng-ningxin
• [ ] submit task to OpenPAI cluster failed #4022 @SparkSnail
• [ ] #3978 @SparkSnail
• [ ] Got Segmentation fault when running amc_search.py #4000 (to check whether there are any version limits) @linbinskn
• [ ] View an experiment with experiment path. #3953 @SparkSnail to verify and close the issue
• [ ] AutoCompress with level pruner leads to runtime error #3948 @J-shang
• [ ] SPOS hardware-aware nas multi_trial.py don't work #4198 @JiahangXu

delayed

• [ ] classification @jonsnows ~Aug 26~
  • [x] investigate search space and common models and chose one to start.
  • [x] review meeting for investigate. Aug 30
  • [ ] implement NAS (run on openpai ne cluster). ~Sep 8~ Sep 10
  • [ ] review meeting for result(demo). ~Sep 15~ ~Sep 22~ Sep 29
  • [ ] improvement and PR(code and doc)
• [ ] object detection @QuanluZhang ~Aug 30~ ~Sep 6~ ~Sep 8~ ~Sep 9~ ~Sep 15~
  • [x] investigate search space and common models and chose one to start.
  • [x] review meeting for investigate. Sep 16
  • [ ] implement NAS, baseline model result
  • [ ] review meeting for result(demo).
  • [ ] improvement and PR(code and doc)

[acured]

Stretch goals

NAS

• [ ] P2 - refactor nas examples @ultmaster
• [ ] P2 - support debug mode for NAS experiment @ultmaster
• [ ] P0 - Add examples for NAS-Bench-series @ultmaster
  • [ ] P2 - 301
• [ ] P2 - Evaluate existing strategies (multi-trial) on NAS benchmarks @ultmaster
• [ ] P0 - Support one system-level optimization: CSE optimization to deduplicate computation @hzhua pr #3202
• [ ] P0 - Improve training service to support CSE optimization @hzhua #3897
• [ ] P2 - TensorFlow support: graph converter and support inline mutation API @liuzhe-lz
  • [ ] add raw framework
  • [ ] add example 7.12
• [ ] P0 - Migrate FBNet from NAS v1.0 to Retiarii framework @Ali
• [ ] P2 - AutoDropout @skeletondyh 7.16
• [ ] P1 - Evaluate one-shot strategies on NAS benchmarks @ultmaster
• [ ] P1 - Further improve the logic of graph generation and code generation (Further integrated with TorchScript) @QuanluZhang
• [ ] P1 - Test classic nas tuners on NNI NAS benchmark and report evaluation results @ultmaster
• [ ] P2 - Migrate Cream from NAS v1.0 to Retiarii framework @ yuhangchen
• [ ] P2 - refactor of Retiarii execution engine @ultmaster
  • [ ] support local debug mode for pure-python execution engine
  • [ ] support weight transfer from supernet to submodel
  • [ ] support export of top model for pure-python execution engine
  • [ ] P2 - refactor of "self._cur_samples" in mutators
  • [ ] P2 - support validation for other strategies
• [ ] P1 - Review strategy/experiment/rest stop condition @QuanluZhang
• [ ] P1 - support graph-based NAS algorithms @ultmaster @skeletondyh
• [ ] P1 - Deal with strategy failure
• [ ] NAS runtime improvement Sep 28
  • [ ] provide option to keep webui when experiment is done (add arg in run()) @QuanluZhang ~Sep 15~ ~Sep 24~
  • [ ] support hyper-parameter figure on webui for retiarii experiment @QuanluZhang ~Sep 15~ ~Sep 24~
  • [ ] support tuner_no_more_trial for retiarii experiment @QuanluZhang ~Sep 15~ ~Sep 24~

Model Compression

• [ ] P0 - support iterative pruning @J-shang 6.18 review meeting (done) / 6.23 sample / 6.25 7.14
  • [ ] P0 - implement model compression 2.0, include 8 maskers (basic pruners) 7.9 #3917
  • [ ] P0 - improve pruners to more faithfully reproduce the papers @J-shang
  • [ ] P2 - Refactor compress() in IterativePruner, unify support AGP, ADMM. (#3669 )
  • [ ] P2 - Add more criteria(strategy, i.e., acc drop toleronce) (discussion needed)
• [ ] P1 - MixedMaskerPruner @J-shang
• [ ] P2 - Refactor of model graph generation (as an independent component) @zheng-ningxin

Training service & NNI manager && nnictl

• [ ] P2 - hybrid integration test on windows @SparkSnail
• [ ] P2 - hybrid integration test with azure storage @SparkSnail
• [ ] P2 - support max_trial_duration for NNI NAS
• [ ] P2 - fix security issues @liuzhe-lz
• [ ] P2 - improve the usage of shared storage for remote training service, e.g., leave storage mount to user (discussion) @acured
• [ ] P2 - improve pipeline integration test yaml (align script and dependences) (investigate) @liuzhe-lz (5 days)
• [ ] P2 - fix ut macos timeout, maybe nnimanager tests, add back some removed tests @liuzhe-lz (5 days)
• [ ] P2 - reduce wheel package size (check) @liuzhe-lz (3 days)
  • [ ] refactor some modules as plugin module (install only when used)
• [ ] P2 - review and improve tests & doc for shared storage, experiment management, jupyterlab, training service
  • [ ] experiment management @liuzhe-lz (3 days)
  • [ ] jupyterlab @liuzhe-lz
  • [ ] training service Doc
  • [ ] training service UT @acured Aug 13
  • [ ] common
  • [ ] kubernetes
  • [ ] local
  • [ ] remote
  • [ ] reusable
    • [ ] shared storage @acured Aug 13 #4068
    • [ ] enviroments
  • [ ] remove HDFS doc from pai doc
• [ ] P0 - (discussion needed) design an API to share data loading/preprocessing among trials (#3294 )
  • [ ] P2 - implementation
  • [ ] P0 - Make reuse default @SparkSnail
  • [ ] P1 - support storage service for different training services, review and implementation
  • [ ] P1 - add more test for remote mode training service
  • [ ] P2 - Frameworkcontroller support @SparkSnail
• [ ] P0 - fix path escape error in windows (issue #3869) @liuzhe-lz @acured @J-shang #3898
  • [ ] add back removed ut
• [ ] P2 - Mask confidential config fields (depends on prev items) @liuzhe-lz
• [ ] P2 - nnictl experiment export improvements @J-shang
• [ ] P2 - internal tuner APIs for improving tuner debuggability, allow debugger tool @liuzhe-lz
• [ ] P2 - nnictl auto-completion @liuzhe-lz
• [ ] P2 - Dependency-aware supported in iterative pruner (discussion needed)
• [ ] P2 - Add full test (l2filter, Taylor, NetAdapt, SimulatedAnnealing, AutoCompress, AMC, Sensitivity, ADMM, Lottery Ticket)
• [ ] P1 - reuse mode: ADL kubeflow support @SparkSnail
• [ ] P2 - add/del tag of experiments (pending support from rest server)
• [ ] UI: support experiment resume/view/stop on webUI
• [ ] P0 - (discussion needed) how to support specifying initial configs (#3801 ) @ultmaster
• [ ] P2 - (discussion needed) support constraints on hyper-parameters @ultmaster
• [ ] P2 - handle all reject errors @J-shang
• [ ] refactor exp config v1 and v2, add validation for v2 @liuzhe-lz Sep 10
  • [ ] make python package yaml >= 5.4, double check dependency, and run pipelines
  • [ ] add test for scientific annotation (e.g., 1e-4) in search space, issue #4080
• [ ] P0 - frameworkcontroller reuse support @SparkSnail Aug 23 #4104

WebUI

• [ ] P0 - Adaptive UI (height) for high resolution, current window for high resolution need improvement @junzhuzhang
  • [ ] overview page: blank at the bottom
  • [ ] align issue in overview page
  • [ ] support searching customized string in the search box

Hyper-parameter tuning

• [ ] P1 - Improve HPO benchmarks, add more concrete conclusions
  • [ ] add one benchmark and one search space 7.14 #3925
• [ ] P1 - support nested search space @kvartet 6.28 discussion meeting
• [ ] P1 - Improve hyperband, PBT with shared storage (dependency on shared storage implementation) @acured
• [ ] P1 - upgrade cuda, config_space, docker image @liuzhe-lz 7.21 #3976
• [ ] P1 - grid search does not support _name for nested search space (update doc) @kvartet
• [ ] P1 - evolution tuner does not mutate sub search space for nested search space @kvartet

jupyter

• [ ] jupyter restart, node process is not stopped.
• [ ] jupyter does not show error message of node process.
• [ ] build jupyterlab-extension as option.

Pipeline

• [ ] P0 - support python 3.9 in pipeline (wait for pytorch upgrade from 1.6 to 1.9) @ultmaster #3881
  • [ ] P2 - write ipc test
• [ ] P0 - fix and improve pipeline
  • [ ] nnimanager ut, nni-ci-gpu-03 failed, maybe due to this machine's environment, cannot be executed through daemon, need reimage -> create a new vm

Doc

• [ ] zh Doc translate.
• [ ] P2 - update NAS API doc to add more supported evaluators @ultmaster
• [ ] doc and demo improvement
  • [ ] end2end, based on comparison of different tuners (insights from benchmark, provide reasonable result)
  • [ ] demo how to share data, how to use shared storage, mainly refactor doc (view experiment from shared storage)

[QuanluZhang]

Bugbash manager @liuzhe-lz (check other prs)

NAS

• [ ] (need review) fix benchmark @ultmaster tester: @QuanluZhang
  • [ ] unify architecture format. Aug 20 #4090
  • [ ] provide simple API for query. Aug 20 #4090
  • [ ] support benchmark end2end (test strategies' performance) #4205
• [ ] support serialization @ultmaster #4172 tester: @liuzhe-lz
• [ ] review and improve nas tests stage 1 @ultmaster. #4097 tester: @liuzhe-lz
• [ ] improve quick start (notebook) @QuanluZhang. #4094 tester: @ultmaster
  • [ ] replace example, update mnist example.
• [ ] examples for image classification, object detection, audio (show reasonably good performance) tester: @ultmaster
  • [ ] transformer @skeletondyh #4222
• [ ] deal with pytorch lightning onnx error for NAS visualization @ultmaster, tester: @QuanluZhang

Model Compression

• [ ] support compression v2 @J-shang #3917 #4074 #4089 #4116 #4178 #4125 tester: @Fiascolsy
  • [ ] tests
  • [ ] (bug bash)doc tutorial
• [ ] support per-channel quantization @linbinskn #4137 tester: @J-shang
• [ ] end2end examples for image classification, object detection, audio, transformer (show reasonably good performance) tester: @J-shang
  • [ ] image classification (MobileNet) @xiaowu0162 #4102

HPO

• [ ] refactor Hyper-opt based algorithm - random tuner. @liuzhe-lz #4118 tester: @ultmaster

nnictl & nnimanager & training service @SparkSnail @liuzhe-lz @acured

• [ ] experiment config doc update @J-shang #4131 tester: @linbinskn
• [ ] improve doc: clarify the difference between training service storage and shared storage @J-shang #4217 tester: @linbinskn

WebUI

• [ ] bug fix - Improve trial intermediate result graph. @Lijiaoa #4171 tester: @SparkSnail
• [ ] bug fix - webui prompts error message. @Lijiaoa #4164 tester: @SparkSnail

regression test needed, or bugs reported during the iteration by external users (need test and close the issues)

• [x] #4077, #4083 Restful server started failed @acured , tester: @liuzhe-lz
• [ ] nni==2.3 tensorboard uninstall then install get version erro,how solve it? #4087 @SparkSnail , resolver&tester: @liuzhe-lz
• [ ] Exponential Notation is not allowed anymore #4080 @QuanluZhang , tester: @linbinskn
• [x] #4158 @acured , tester: @linbinskn
• [ ] assert len(set(num_channels_list)) == 1 #4160 @J-shang , tester: @J-shang
• [ ] Task is always in a waiting state in the remote machine #3905 (not an 2.4 regression but a 2.1 regression) @acured , tester: @SparkSnail
• [ ] NNI installation issue with ConfigSpace-0.4.7 #3909 (not sure whether this is a regression or not) @liuzhe-lz , tester: @liuzhe-lz
• [ ] Runtime Error during L1FilterPruner (and other one-shot pruners) #3944 @zheng-ningxin , tester: @J-shang
• [ ] submit task to OpenPAI cluster failed #4022 @SparkSnail , tester: @QuanluZhang
• [ ] #3978 @SparkSnail , tester: @SparkSnail
• [ ] AutoCompress with level pruner leads to runtime error #3948 @J-shang, tester: @J-shang
• [ ] SPOS hardware-aware nas multi_trial.py don't work #4198 @JiahangXu , tester: @ultmaster

[liuzhe-lz]

BUG BASH (Target date Friday 10/22)

PRs seemingly need to test:

model compression

• chenbohua3
  • [x] @linbinskn #3992
  • [x] @linbinskn #4038
  • [x] @linbinskn #4039
  • [x] @linbinskn #4084
  • [x] @linbinskn #4113
  • [x] @linbinskn #4127
  • [x] @linbinskn #4137
    • [x] Problem found, @linbinskn submitted a fix PR #4252
• Fiascolsy
  • [x] @J-shang #4184 #4225
  • [x] @J-shang #4245
• J-shang
  • [x] @Fiascolsy #3917 #4074 #4089 #4116 #4178
  • [x] @Fiascolsy #4236
• linbinskn
  • [ ] @J-shang #3990 Waiting for #4252
  • [x] @J-shang #4049
  • [x] @J-shang #4148
  • [x] @J-shang #4163
• xiaowu0162
  • [x] @J-shang #4102
• zheng-ningxin
  • [x] @J-shang #4078
  • [x] @J-shang #4128
  • [x] @J-shang #4144
  • [x] @J-shang #4173
  • [x] @J-shang #4241

    nas

• hzhua
  • [x] @QuanluZHang #4075 #4086 #4098
• JiahangXu
  • [x] @ultmaster #4092
  • [x] @ultmaster #4214
• QuanluZhang
  • [x] @ultmaster #4094
• ultmaster
  • [x] @QuanluZhang #4090 #4205
  • [x] @zheng-ningxin #4154
  • [x] @liuzhe-lz #4172

    other

• JunweiSUN
  • [x] @liuzhe-lz #4108
• Lijiaoa
  • [x] @SparkSnail #4164
  • [x] @SparkSnail #4171
• liuzhe-lz
  • [x] @QuanluZHang #4147
  • [x] @QuanluZHang #4177
  • [x] @ultmaster #4118 #4238
• SparkSnail
  • [x] @acured #4104
  • [x] @acured #4199

    contributors

• alibaba
  • [ ] (no environment) #4055 @weidankong need you to test, verify and confirm, thanks!
  • [x] @liuzhe-lz #4167
  • [x] @liuzhe-lz #4188
• individuals
  • [x] @Fiascolsy #4121
    • [x] @liuzhe-lz and @Fiascolsy follow up on problems found during tests.
  • [ ] @ultmaster #4153 @QuanluZhang
  • [x] (no need to test) #4161
  • [x] (no need to test) #4208

[liuzhe-lz]

To fix:

• [ ] Tutorial for Python experiment API training service @liuzhe-lz next release
• [ ] lightning trainer doc @ultmaster
• [x] experiment config code default value @J-shang
• [x] HPO benchmark
• [x] Add "experimental" to CGO doc

[QuanluZhang]

• [x] double check gpuIndices, trialGpuNumber, @linbinskn @liuzhe-lz

[cruiseliu]

Model Compression

• New major version of pruning framework (doc)
  • Iterative pruning is more automated, users can use less code to implement iterative pruning.
  • Support exporting intermediate models in the iterative pruning process.
  • The implementation of the pruning algorithm is closer to the paper.
  • Users can easily customize their own iterative pruning by using PruningScheduler.
  • Optimize the basic pruners underlying generate mask logic, easier to extend new functions.
  • Optimized the memory usage of the pruners.
• MobileNetV2 end-to-end example (notebook)
• Improved QAT quantizer
  • support dtype and scheme customization
  • support dp multi-gpu training
  • support load_calibration_config
• Model speed-up now supports directly loading the mask
  • [ ] work item: add ModelSpeedup API reference doc
• Support speed-up depth-wise convolution
• Support bn-folding for LSQ quantizer
• Support QAT and LSQ resume from PTQ

Neural Architecture Search

• NAS benchmark (doc)
  • Support benchmark table lookup in experiments
  • New data preparation approach
• Improved quick start doc
• Experimental CGO execution engine (doc)

Hyper-Parameter Optimization

• New training platform: Alibaba DSW+DLC (doc)
• Support passing ConfigSpace definition directly to BOHB (doc) (thanks to khituras)
• Reformatted experiment config doc
• Added example config files for Windows (thanks to politecat314)
• FrameworkController now supports reuse mode

Fixed Bugs

• Experiment cannot start due to platform timestamp format (issue #4077 #4083)
• Cannot use 1e-5 in search space (issue #4080)
• Dependency version conflict caused by ConfigSpace (issue #3909) (thanks to jexxers)
• Hardware-aware SPOS example does not work (issue #4198)
• Web UI show wrong remaining time when duration exceeds limit (issue #4015)
• cudnn.deterministic is always set in AMC pruner (#4117) thanks to mstczuo

And...

• New emoticons!