v2.0 Release Plan

[QuanluZhang]

This is the draft plan for the upcoming iteration in Oct ~ Nov, it's a 4 weeks iteration.

Release Plan for v2.0

• Release manager: @liuzhe-lz
• Feature freeze date: ~11.20~ ~11.27~ 11.30
• Code freeze and demo date: ~12.4~ ~12.8~ ~12.11~ ~12.14~ 12.16
• Release date: ~12.21~ ~12.28~
  • alpha 2: 12.28
  • alpha 3: ~12.30~ 1.4
  • rc: ~1.4~ 1.6

  • release: 1.11

    Let's try to adopt the legend of annotations:

Mark	Description
🏃‍♂️	work in progress
👪	work under review
✋	blocked task
💪	stretch goal for this iteration
🔴	missing issue / pull request reference
🔵	more investigation and survey required
⚫	under discussion within the team
⬛	a large work item, larger than one iteration

Epic

• [ ] Improved NAS framework (Retiarii) on NNI, 40 days

  • [x] graph converter for PyTorch code, 10 days，#2938, dev owner: @QuanluZhang
  • [x] execution engine (with integration), 5 days, #2939, dev owner: @ultmaster
  • [x] mutation, 6 days, #2941, dev owner: @liuzhe-lz
  • [x] User interface, 5 days @liuzhe-lz ETA: ~11/18~ ~11/20~ (PR #3081 merge +1d) PR #3111
    • [ ] weight sharing support @QuanluZhang
  • [x] cross-graph optimizations, 10 days, #2940, dev owner: @hzhua ETA to be decided: PR #3105
  • [x] integration (first end-to-end), 5 days, #2942, dev owner: @QuanluZhang ETA: ~11/13~ ~11/16~ after user interface (review: @liuzhe-lz @ultmaster )
  • [x] search space (P0 Quanlu 1, P0 Yuge 1, P1 others best efforts) support and test, 10 days, #2943, dev owners: @ultmaster /@QuanluZhang ETA: P0 11/20; P1: 11/27
  • [ ] integration tests, 5 days, #2944, dev owner: @liuzhe-lz ETA: (depends on #2943's P0 API) ~11/27~ 11/2 (after feature freeze)
  • [x] Weight sharing, 10 days @QuanluZhang TODO: before 11/17 Quanlu drive the possible implementation plan; after 11/17, Yuge take over design and implementation. ETA: 11/27
  • [ ] P2 - (blocked by NAS search space structure) support NAS Once-for-all alog on NNI, 5 days, @colorjam #2955

• [x] Support heterogenous training service @SparkSnail PR #3097 (review: @squirrelsc @liuzhe-lz )

  • [x] Support remote
  • [x] Support azureml
  • [x] Support local
  • [x] Support PAI
  • [x] doc 11.25
  • [ ] refactor code for review 11.26
  • [ ] IT pipeline (on going)

• [ ] Release materials @scarlett2018 /@liuzhe-lz (after feature freeze 11/27, target end date 12/11)

  • [ ] release process, 1-2 week @liuzhe-lz
  • [ ] doc, article, and demo 1 week @scarlett2018
  • [ ] Release press (demo new features) @scarlett2018
    • [ ] Upload video

Features

• [x] Reorganize NNI python package, package hierarchy, source code directory hierarchy - dev owner: @liuzhe-lz
  • [x] P0 - able to install and run, 3 days, ETA: 10/15
  • [x] add UT pipeline test, 7 days, ETA: 10/26 PR #3037 review: @chicm-ms @SparkSnail
  • [x] IT pipelines PR #3088 review: @SparkSnail @chicm-ms
• [ ] P2 - Add one NAS NLP benchmark, 7 days - dev owner: @98may (@ultmaster offers help), #2954 ETA: ~11/17~ ~11/20~ 11/27
• [x] P0 WebUI, 9 days
  • [x] backend support of experiment management, @J-shang PR #3081 review: @SparkSnail @liuzhe-lz
  • [x] add experiment management page 6 days, @lijiaoa (demo done) PR #3127
  • [x] log UI refactor 3 days (@Lijiaoa ) 11/6 PR #3070 ready for review
  • [x] #2937 overview refactor @yangou1988 PR #3046 (review: @ultmaster @corlorjam )
    • [x] Last update 11/25 PR #3123
• [x] P1 - model compression / graph_utils supports pytorch 1.6 #2877 #2879 @chicm-ms ~11/6~ PR #3076 (review @QuanluZhang @Zheng-ningxin )
  • [ ] Fix test case @ultmaster
• [x] P2 - add fpgm to agp /other pruning schedule (add more benchmark results) #2853 , 2 day, @chicm-ms ETA: ~11/17~ 11/19 PR #3110
• [ ] P2 - refactor experiment config file, @liuzhe-lz draft PR #3138
  • [ ] 2 days design & discussion (issue #3087 )
  • [ ] 3 days implementation ETA: 11/27
• [ ] P2 - Tuning algorithm's meta data are located in three different places, plan to unify them, design, depends, @chicm-ms , 5 days PR #3134

Tasks

• [x] Port doc to rst and fix links, 2 day evaluation, @ultmaster

• [x] Unify the names of trail job id; @J-shang 9/16 , pr ~#2896~ #3053 (review: @chicm-ms @QuanluZhang )

• [x] use NNI's flops counter (refactor the counter first) @colorjam PR: #3048 reivew: @chicm-ms @QuanluZhang @ultmaster

• [x] Bug fix of experiment resume, @J-shang 1 day (evaluated by quanlu)

• [x] check experiment duration under the tuner_no_more_trial state (Issue #2758 ), @J-shang 1 day PR: #3043 @liuzhe-lz @QuanluZhang

• [ ] AML scheduling (Issue #3014) @SparkSnail 1 day ~PR #3034 review: @QuanluZhang @chicm-ms ~

• [x] HyperBand refactor @J-shang 3 days PR: #3040 review: @QuanluZhang @colorjam

• [x] Add algorithm import shortcut @liuzhe-lz 3 days draft PR #3139

• [x] AdaptDL integration test @SparkSnail

• [x] Quantization export @linbinskn

• [ ] update API reference for model compression in doc

• [x] Import shortcut @ultmaster

• [x] Pipeline (fix ut numpy dependency, check it) @ultmaster

• [x] Fix NNI manager warning @J-shang

Stretch Goals & Deferred Items

• [ ] tensorboard backend support, 5 days @J-shang (@SparkSnail help)

• [ ] multi-objective optimization algorithm, #2525 , 5 days - @ultmaster

• [ ] Cream2NNI @hwpengms , pr #2705 , under refactoring (review @QuanluZhang )

• [ ] Update TF model compression: support checkpoint, 3 days @liuzhe-lz

• [ ] P2, 2.0-candidates, under design discussion support post training quantization speedup, 14 days - dev owner: @linbinskn @J-shang

• [ ] P2 Update node/python JSON handling, 3 days - @liuzhe-lz

  • [X] survey node/python JSON handling
  • [ ] fix the JSON parse twice issue, 1 day
  • [ ] Flatten nested JSON in RPC protocol, 5 days

• [ ] FBnetV3 tuning algorithm, #2591 , #2357 , 7 days - dev owner: @J-shang

• [ ] HyperBand further refactor @J-shang

• [ ] Merge .config and db.experimentProfile.param @J-shang

• [ ] Support code coverage check, add this check to IT, 7 days @chicm-ms

• [ ] test classic nas tuners on NNI NAS benchmark and report evaluation results, ? days, @98may

• [ ] Search trial by parameter (issue #3024) @sharonxingyufan @Lijiaoa

• [ ] Bump dependency version, @liuzhe-lz 5 days

Backlog

• [ ] Tutorial to use tuner directly instead of using nni manager (like hyperopt?) (maybe @liuzhe-lz next release)
• [ ] NAS visualization improvements: refactor graph representation @ultmaster (v2.1 candidate)
• [ ] AutoML for data augmentation (image and NLP) (research)
• [ ] Write search space pruning with flops in a separated component (research)

[liuzhe-lz]

Bug Bash

• [x] Document convertion @ yiru
• [x] Retiarii
  • [x] Graph gen @ultmaster
  • [x] Weight sharing @QuanluZhang
  • [ ] Dedup @colorjam
  • [x] end-to-end @colorjam
• [x] Heterogenous training service @liuzhe-lz
• [x] NAS NLP benchmark @ linbin
• [x] Web UI new layout @ yiru
• [x] Experiment management @SparkSnail
• [x] Model compression
  • [x] PyTorch 1.6 support @J-shang
  • [x] FPGM for AGP @J-shang
  • [x] Flops counter @ linbin
• [x] Launch from Python @J-shang
  • [x] fix (note: add dispatcher log) PR #3245
• [x] New configuration @ultmaster
  • [ ] fix
• [x] Customized tuner install @liuzhe-lz
• [x] Experiment resume bugfix @QuanluZhang
  • [ ] fix
• [x] Experiment duration under TUNER_NO_MORE_TRIAL @98may
• [x] HyperBand refactor @ ayan
• [x] ADL @SparkSnail
• [x] Quantization export @QuanluZhang
• [x] NNI manager warning @QuanluZhang
• [x] Check other PRs not listed in tasks @98may
• [x] Kill trial @liuzhe-lz
• [x] Logging (check if NNI manager, dispatcher, trial log contains logging and printing output, including multiprocess scenario) @ultmaster
• [x] nnictl save , load , log commands @98may
• [x] quantization bug (PR #3160) @J-shang
• [x] Cream @ultmaster

Documentation

• [x] Retiarii @QuanluZhang
  • [ ] ETA for v1 @QuanluZhang 12/30; Reviewers: @colorjam @ultmaster @kvartet @scarlett2018
• [x] Heterogenous
  • [ ] [heterogenous --> hybrid] ETA for the 12/31 @kvartet @scarlett2018
• [x] Launch from Python PR ready @J-shang #3217 ; Reviewers: @liuzhe-lz @QuanluZhang @kvartet; ETA for review: 12/31;
• [x] New experiment config @liuzhe-lz #3222; Reviewer: @SparkSnail ; TODO: @liuzhe-lz fix reviewer comments; + @kvartet (comments: there are lots of TODO in the doc); ETA for doc update: 12/31;
• [x] New custom algorithm register mechanism ask @chicm-ms (@J-Shang) PR #3241
• [x] Translation @kvartet; ETA for the PR: 12/31;
• [x] README @scarlett2018 ETA for 1/4

Document Update & Regression Test

• [ ] HPO
  • [x] mnist-{a,b}... , networkmorphism, autogbdt, gasquad @QuanluZhang
  • [x] other mnist @ linbin
  • [x] other trials @98may
  • [ ] fix: Update doc, remove paiYarn
  • [ ] custom tuner examples (TODO)
  • [x] System autotuning @QuanluZhang > @ xiaotian
• [x] NAS @ultmaster
• [ ] Model Compression
  • [x] Regression @colorjam
  • [x] Speedup @ ningxin
• [x] Training Service @J-shang
  • [x] kubeflow @ linbin & @liuzhe-lz
• [x] Reference @liuzhe-lz
• [x] Overview @ yiru
  • [x] Install from source code doc
• [ ] Others
  • [x] Feature Engineering @98may

Pipelines

[liuzhe-lz]

Bugs Found in Bugbash

• [x] Quantization BNN & Dorefa @ linbin PR #3247
• [x] Configuration v2 error handling @ultmaster PR #3248
• [ ] Retiarii output @QuanluZhang update doc
• [x] Python experiment termination
  • [x] Linux @liuzhe-lz
  • [x] Windows @J-Shang
• [x] OpenPAI windows mountpoint check @liuzhe-lz
• [x] Remove PAI yarn doc @SparkSnail
• [x] Reuse in remote training service doc @SparkSnail
• [x] Update hybrid doc that windows does not support local @SparkSnail
• [x] Web UI doc update @lijiaoa #3229
• [x] Install from source doc @liuzhe-lz
• [x] Change quickstart example to torch @liuzhe-lz
• [x] Change experiment management page order @lijiaoa
• [x] nnictl trial ls timestamp @J-Shang #3235
• [x] hybrid v2 config @QuanluZhang
• [x] add new config format example file in mnist-tf and mnist-pytorch examples @ultmaster

[liuzhe-lz]

• [ ] Retiarii strategy doc @QuanluZhang
• [ ] Retiarii output doc @QuanluZhang
• [ ] Release note @ yiru
• [ ] Pipeline @liuzhe-lz