xorbitsai / xoscar

Python actor framework for heterogeneous computing.
https://xoscar.dev
Apache License 2.0
91 stars 21 forks source link

FEAT: Impl collective communication interface #46

Closed ChengjieLi28 closed 1 year ago

ChengjieLi28 commented 1 year ago

Related #22

Tested on xorbits.

In the case of worker number 5 for tpch 100, broadcast is much more slower than copy_to. Maybe when cluster scale is very large, collective communication may speed up. But I do not know now. See issue: https://stackoverflow.com/questions/26558189/mpi-broadcast-very-slow

Check code requirements

codecov[bot] commented 1 year ago

Codecov Report

Merging #46 (cac0e28) into main (c3ac228) will increase coverage by 0.34%. The diff coverage is 95.51%.

@@            Coverage Diff             @@
##             main      #46      +/-   ##
==========================================
+ Coverage   93.66%   94.00%   +0.34%     
==========================================
  Files          42       47       +5     
  Lines        3361     3671     +310     
  Branches      672      708      +36     
==========================================
+ Hits         3148     3451     +303     
- Misses        141      145       +4     
- Partials       72       75       +3     
Flag Coverage Δ
unittests 93.87% <95.51%> (+0.35%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
python/xoscar/collective/utils.py 63.63% <63.63%> (ø)
python/xoscar/collective/core.py 95.57% <95.57%> (ø)
python/xoscar/collective/process_group.py 96.71% <96.71%> (ø)
python/xoscar/backends/communication/ucx.py 94.61% <100.00%> (+2.27%) :arrow_up:
python/xoscar/collective/__init__.py 100.00% <100.00%> (ø)
python/xoscar/collective/common.py 100.00% <100.00%> (ø)

... and 2 files with indirect coverage changes