FEAT: Slurm Deployment For Xorbits

fengsxy commented 9 months ago

What do these changes do?

In the past, if the user need to deploy xorbits on the slurm, they need to construct bash code by themselves. And it's so hard to debug for the slurm systems, because the output file should only output in the shared file. So with the help from xorbits, I have constructed Class which is named Slurm, which could generate the slurm.sh and sbatch it automatically. What's more, the docker with slurm cluster test environment has been completed.

Related issue number

Fixes #315

Check code requirements

[x] add a class that could generate slurm batch code
[x] add the function which could sbatch the code
[x] add the function when the programming end just cancel the xorbits on slurm
[x] add a environment that ensure the slurm cluster with xorbits
[x] add test on generate slurm batch code
[x] add test on sbatch script and get the true address and sbatch
[x] completed related documents
[x] Ensure all linting tests pass

ChengjieLi28 commented 9 months ago

Please fix your python lint firstly. Refer https://doc.xorbits.io/en/latest/development/contributing_codebase.html#pre-commit to use pre-commit to format your code before git commit.

codecov[bot] commented 9 months ago

Codecov Report

Merging #719 (a0da962) into main (b320ca3) will increase coverage by 11.03%. The diff coverage is 100.00%.

@@             Coverage Diff             @@
##             main     #719       +/-   ##
===========================================
+ Coverage   82.55%   93.58%   +11.03%     
===========================================
  Files        1058     1059        +1     
  Lines       79780    79781        +1     
  Branches    16504    16504               
===========================================
+ Hits        65861    74663     +8802     
+ Misses      11645     3443     -8202     
+ Partials     2274     1675      -599

Flag	Coverage Δ
unittests	`93.47% <100.00%> (+11.05%)`	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
python/xorbits/deploy/slurm/__init__.py	`100.00% <100.00%> (ø)`

... and 171 files with indirect coverage changes

xorbitsai / xorbits