OpenPAI Backlog

[scarlett2018]

This issue is a long term backlog for planning and discussion. Please feel free to add "the top in your mind" to this issue directly.

• [ ] 2021 Feb Release Plan #5253
• [ ] 2021 March Draft Plan #5346

Last Updated on 02/01/2021

Top Focus Scenario

• Support HiveD Management experience
• AI Optimized Workload E2E support
  • use case study for internal & external typical workloads

---

Multi-Cloud/Multi-Cluster

• Hybrid installation
• multi/cross cluster management
• multi-cluster job scheduling

Job management

• [ ] Favorite Jobs

AutoScaler

• new added node's provisioning

Engineering Excellent

• CI/CD
• Build process

More examples for the current examples

Add user team support

• Jupyter Notebook support

• FC detect the preemption, gc, eviction, etc. @yqwang-ms

• FC rescale Framework @yqwang-ms

• Update the official docker image (comparing with deepo) @hzy46

Installation experience

• Topology planning
• Installation pre-check (especially for installation that start from non-clean env)
• Azure installation practices
  • SKU support for complex job types
  • Network plugin deployment
  • [ ] migrate python2 to python3 https://github.com/microsoft/pai/issues/5042
  • [ ] A kube runtime plugin to export parameters to env #4636

[fanyangCS]

https://github.com/microsoft/pai/issues/3872

[scarlett2018]

History Backlog Info Backup, only for reference.

Brainstorming on 2020/09/10

~[Planned in #4898 ] Cell as sku in hived scheduler.~

• [ ] Refactor duplicate deployment code.
• [ ] Refactor job submission form.

~[Planned in #4898] Support dynamic sku types in different vc.~

• [ ] Remaining works in New RestServer Architecture: RestServer -> DB -> ApiServer #4651

@hzy46 : ~[Planned in #4898] pod/event watcher to support #4649~ ~[Planned in #4898 ] multi cluster management detailed design & review~

• merge previous doc change #4828 #4821
• db gcer
• archive job (tag) #4453
• dbcontroller optimization items #4849 #4850 #4794
• make DBController tolerant to wrong framework request #4889
• ui to let admin add storage #4806

@suiguoxin: #4789 ~[Planned in #4898] Alert-manager: Kill low-gpu-utilization jobs, tag abnormal jobs, Cordon node with k8s API when GPU GCC Error~ ~[Planned in #4898] Job tags : DB table, rest-server API, web-portal refactor~

@yiyione: ~[Planned in #4898] Group management page in webportal~ ~[Planned in #4898] VC request management for user and admin~

• [ ] VSCode client bug fix

@debuggy:

• marketplace related items https://github.com/microsoft/openpaimarketplace/issues/60 ~[Planned in #4898]Grammar check~
  • [ ] NNI job integration -- Feature Engineering
  • [ ] add info for items in submit job page
  • [ ] add "Upload" feature for dataset, set upload standard and workflow
• [ ] job submit refactor
  • [ ] new submit workflow design
  • [ ] team storage
  • [ ] sku
• [ ] job detail work part 2
  • [ ] job retry more detail
  • [ ] show events in job detail page

Per Task Retry History

• [ ] FC generates Task history snapshots (mocked Task CRD object with UID) @yqwang-ms
• [ ] Fluentd + DB to collect above snapshots @hzy46
• [ ] WebPortal expose the Per Task Retry History (and failure counts grouped by failure type within one job?) @debuggy

Brainstorming on 5/14 - last updated at 5/28

• Candidate ideas about PAI experience improvements #3338

Multi-Cloud/Multi-Cluster
@hzy46

• Hybrid installation
• multi/cross cluster management
• multi-cluster job scheduling

AutoScaler @ydye

• new added node's provisioning

Support for large scale cluster

• [ ] (June) set up latency benchmark report for 100 virtual nodes --> 1500 virtual nodes * 8 job @ydye
• [ ] based on the stress and performance test result, analyze and plan for improvements

GPU Utilization

• [ ] (June) improve training cost ROI for EDU scenario @hzy46
• [ ] improve current internal training workload utilization @yiyione

• [ ] (June) Performance improvement for DB (Use Database as History Server #4610) @hzy46

More examples for the current examples

Add user team support

Surfacing more backend error to users in Job Details Page #4649 @qfyin ,@yqwang-ms potential failure case: storage mount failed

• [ ] Surfacing more backend error to users (GCC error, out of storage, etc.)

• [ ] K8S Pod keep waiting in creating container, such as ErrImagePull #3572

• [ ] GPU Fairness Usage @Binyang2014

• [ ] Usability improvement based on user study and feedback @yangou1988

• Service plug-in (user customized) @hzy46

• Jupyter Notebook support @hzy46

• Runtime interface @Binyang2014

• HiveD Improvements (i.e. aware per GPU bad, etc.) @yqwang-ms @zhypku

• FC detect the preemption, gc, eviction, etc. @yqwang-ms

• FC rescale Framework @yqwang-ms

• Update the official docker image (comparing with deepo) @hzy46

Installation experience

• Topology planning
• Installation pre-check (especially for installation that start from non-clean env)
• Azure installation practices
  • SKU support for complex job types
  • Network plugin deployment
  • Support for new hardware
  • [ ] migrate python2 to python3 https://github.com/microsoft/pai/issues/5042

• [ ] A kube runtime plugin to export parameters to env #4636

These are incomplete items from v0.17

• [ ] Grafana metrics link does not have time interval

• [ ] Skip intra communication for elastic job (Binyang https://github.com/microsoft/pai/issues/3708)

• [ ] Bring back debug mode (Binyang https://github.com/microsoft/pai/issues/3643, https://github.com/microsoft/pai/issues/2210)

• [ ] replace disk cleaner with k8s pod eviction policy (https://github.com/microsoft/pai/issues/3340, https://github.com/microsoft/pai/issues/3765)

• [ ] When job is waiting, show the detailed waiting reason somewhere in the job page. (https://github.com/microsoft/pai/issues/3847)

• [ ] Archive logs to system provided storage

• [ ] Exec to job containers on webportal

• [ ] persist service log (https://github.com/microsoft/pai/issues/2129)

• [ ] differentiate opportunistic job and normal jobs in VC usage page.

• [ ] improve user group adding to VC experience

• [ ] improve HiveD configuration experience

• [ ] Show job priority in webportal

• [ ] introduce a new job state "initialization" (https://github.com/microsoft/pai/issues/4141)

• [ ] open ssh connection in browser

• [ ] ssh key management and simplifying user behavior of ssh plugin

• [ ] open ssh connection in browser

• [ ] favorite jobs https://github.com/microsoft/pai/issues/2749

• [ ] actual resource usage of a job https://github.com/microsoft/pai/issues/4695

• [ ] Favorite Jobs

---

Engineering Excellent

• CI/CD
• Build process

---

Low Priority Postponed items

• (Postponed from Aug 2020 release) Elastic DL and Job level scale up and scale down - @yqwang-ms check with Ming/NNI and other integration scenarios - TODO: summarize the to-the-customer level scenarios supports