microsoft / DLWorkspace

Deep Learning Workspace
Other
202 stars 75 forks source link

VC resource quota proportional update #1215

Closed Anbang-Hu closed 4 years ago

Anbang-Hu commented 4 years ago

It has been a pain for us to manually update the resourceQuota in vc table when there is quota shifting from a vc to another. This PR

  1. exposes a rest api /ResourceQuota for admin to update resourceQuota in vc. Resources for GPU sku will be set proportionally to requested GPU quota, while resources for CPU sku wikk be set proportionally to requested CPU quota. The restful API server will invalidate the local cache afterwards, saving us the extra step to restart it. @Gerhut need your help to add a dashboard page for cluster admin to manage VC quota.
  2. add a command in ctl.py to query/update resource quota for vc:
    
    $ ./ctl.py quota show  # brief view on GPU (and CPU)
    vc          Standard_ND24rs(P40)  |
    --------  ----------------------  ---
    platform                      12  |
    vc1                            0  |

$ ./ctl.py -x quota show # detail view (GPU, GPU memory, CPU, memory) gpu:

vc Standard_ND24rs


platform 12 vc1 0

gpu_memory (Gi):

vc Standard_ND24rs


platform 288 vc1 0

cpu:

vc Standard_ND24rs


platform 72 vc1 0

memory (Gi):

vc Standard_ND24rs


platform 1344 vc1 0

$ ./ctl.py quota update --vc_name platform --sku Standard_ND24s --quota 13

coveralls commented 4 years ago

Pull Request Test Coverage Report for Build 3571


Totals Coverage Status
Change from base Build 3570: 0.0%
Covered Lines: 827
Relevant Lines: 874

💛 - Coveralls