NNI 2021 May~June Iteration Planning

[kvartet]

This is the plan for the iteration in May~June, it's a 6 weeks iteration. It has been tracked in the v2.3 Project.

Release Plan for v2.3

• Release manager: @ultmaster
• Feature freeze date: 5.21
• Code freeze and demo date: 5.27
• Branch cut and next release planning date: 6.4
• Release date: 6.9

---

Retiarii & NAS

• [x] P0 - More high-level APIs for users to easily express model space @ultmaster (PR #3481)
• [ ] P0 - TensorFlow support: graph converter and support inline mutation API @liuzhe-lz
• [ ] P0 - NAS visualization improvements: refactor graph representation @98may 5.14
• [x] P1 - Pure-python execution engine @ultmaster 5.7
• [x] P1 - Support RL strategy @ultmaster
• [ ] P1 - Further improve the logic of graph generation and code generation @QuanluZhang 5.20
  • [ ] Further integrated with TorchScript
  • [x] Bug fix (list: #3573)
• [x] new NAS algorithm: Blockwise DNAS FBNet
• [ ] P1 - Support one system-level optimization: CSE optimization to deduplicate computation

Model Compression

• [ ] P0 - Auto shape inference @J-shang taking from @zheng-ningxin
• [ ] P1 - Multi-compressor @J-shang 4.30
• [x] P1 - Auto compressor @J-shang
• [ ] P1 - collaboration with nnmeter @zheng-ningxin
• [x] P1 - module compression pipeline @colorjam
• [x] P1 - Support any tensorflow algorithm @liuzhe-lz 4.30
• [x] Add LSQ quantizer
• [ ] P2 - Add exclude config to AGP pruner (discussion needed)
• [ ] P2 - Dependency-aware pruner with schedule (discussion needed)

Training service & NNI manager && nnictl

• [ ] P0 - Add test for shared storage (PR #3455) @acured 4.30
• [ ] P0 - Make reuse default (needs check) @SparkSnail
• [x] P1 - Update documentation and pipeline to v2 @SparkSnail
• [x] P1 - Remove launching tensorboard from nnictl @SparkSnail
• [x] P1 - Add the output of subprocess (#3141) @acured
• [x] fix v2 aml config and set version check as a warning
• [x] improve NNI manager logging
• [ ] P2 - Hybrid kubeflow support (migrate kubeflow to environment service) @SparkSnail
• [ ] P2 - Mask confidential config fields (depends on prev items) @liuzhe-lz
• [x] P2 - Update training service interface @liuzhe-lz (design review)
• [ ] P2 - nnictl experiment export improvements @J-shang

WebUI

• [x] P0 - Make selected trials consistent in detail table @Lijiaoa 5.7
• [x] P1 - Improve search parameters on trial detail page @Lijiaoa
• [x] P1 - set base url for the web report (#2771) @acured
• [x] add trial stdout button on local mode
• [ ] P2 - Show dict intermediate result in the dialog of single trial intermediate result @Lijiaoa

UX Design

• [ ] P1 - NAS visualization
  • [ ] UX Design - concept design @QuanluZhang / @scarlett2018
  • [ ] UX Design - formal design @yangou1988
• [ ] P2 - Adaptive UI (height) for high resolution, current window for high resolution need improvement @yangou1988 @Lijiaoa
• [ ] Support experiment resume/view/stop on webUI
• [ ] Experiment/Project Management: tag/description

Hyper-parameter tuning

• [x] P1 - HPO benchmark @xiaowu0162
• [x] P1 - DNGO tuner @98may
• [ ] P2 - Improve hyperband, PBT with shared storage (dependency on shared storage implementation) @J-shang

Pipeline

• [x] P0 - add pipeline badge on readme @ultmaster
• [x] P0 - add aml pipeline test @SparkSnail (#3477 draft pr)
• [x] P0 - add hybrid pipeline test @SparkSnail (#3478 draft pr)

Demo

• [ ] v2.2 example and illustration for new features @kvartet
• [ ] Add new demos in more details @kvartet

[ultmaster]

Endgame

Tests

NAS

• [ ] Repeat and Cell in Retiarii high-level APIs (#3481) Quanlu
• [x] Pure-python execution engine (#3605) Quanlu
• [ ] Policy-based RL strategy (#3650) Quanlu
• [x] Nested ModuleList and a special case in list append in Retiarii (#3652) Yuge
• [x] Blockwise DNAS FBNet (#3532) Yuge
• [x] Fix experiment_working_directory in Retiarii config (#3607) Yuge

Model compression

• [x] Auto compression (#3631) Bin
• [x] Model compression e2e example (#3507) (#3688) Ning
• [x] Tensorflow slim pruner (#3614) Ning
• [x] LSQ quantizer (#3503) Bin
• [x] Fix a bug in mask conflict (#3629) Ning
• [x] Fix a bug in model speedup shape inference (#3588) Ning

Training service and Rest

• [x] Support for 3rd-party training service (#3662) Shinai
• [x] Track trial stdout (#3690) (#3653) Yuge
• [x] Prefix URL in RestFUL API (#3648) (#3625) Zhe (Hao)
• [x] Fix an issue in AML V2 config (#3655) Zhe
• [x] Fix syntax error on Windows (#3634) Ning
• [x] Report version check failure as warning (#3654) Yiru
• [x] Fix an issue that causes missing log and metric on Windows when using PyTorch with multithread (#3604)
• [x] Improve NNI manager logging (#3624) Shinai
• [x] Delete redundant code in training service (#3526)

HPO

• [x] DNGO tuner (#3479) Yuge
• [x] HPO benchmark (#3644) Ayan

WebUI

• [x] Improved search parameters on trial detail page (#3651) Yuge
• [x] Make selected trials consistent after auto-refresh in detail table (#3597) Yuge

nnictl

• [x] Delete tensorboard on nnictl (#3613)

Examples

• [ ] Add notebook for tabular data classification in AML (#3599) Quanlu

Pipeline and tests

• [ ] Hybrid and V2 config pipeline (#3648)
• [ ] AML pipeline (#3477)
• [ ] Unpin pytorch_lightning<1.2 (#3598)
• [ ] Fix an issue in documentation of Hybrid training service (#3600)
• [ ] Pin yargs<17.0 (#3608)
• [ ] Update issue bug report template (#3501)
• [ ] Add pipeline badge in README (#3589)

Documentation and example work

• [ ] Promote Retiarii docs and deprecate old NAS framework
• [ ] Convert all v1 examples to v2

[kvartet]

closed as v2.3 has been released.