xorbitsai / xorbits

Scalable Python DS & ML, in an API compatible & lightning fast way.
https://xorbits.readthedocs.io
Apache License 2.0
1.1k stars 67 forks source link

FEAT: MMAP storage backend #674

Closed ChengjieLi28 closed 1 year ago

ChengjieLi28 commented 1 year ago

What do these changes do?

Use mmap as storage backend. Now Xorbits can run the complete TPCH SF100 queries on a single MacOS laptop with 32GB memory.

Future work:

Check code requirements

codecov[bot] commented 1 year ago

Codecov Report

Merging #674 (a949b7a) into main (feacc2e) will decrease coverage by 0.27%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #674      +/-   ##
==========================================
- Coverage   93.59%   93.33%   -0.27%     
==========================================
  Files        1025     1026       +1     
  Lines       79733    79812      +79     
  Branches    16538    16563      +25     
==========================================
- Hits        74625    74490     -135     
- Misses       3430     3665     +235     
+ Partials     1678     1657      -21     
Flag Coverage Δ
unittests 93.22% <100.00%> (-0.27%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
python/xorbits/_mars/deploy/oscar/local.py 97.05% <100.00%> (+0.07%) :arrow_up:
python/xorbits/_mars/storage/__init__.py 69.23% <100.00%> (+2.56%) :arrow_up:
python/xorbits/_mars/storage/mmap.py 100.00% <100.00%> (ø)

... and 25 files with indirect coverage changes