xorbitsai / xorbits

Scalable Python DS & ML, in an API compatible & lightning fast way.
https://xorbits.readthedocs.io
Apache License 2.0
1.1k stars 67 forks source link

ENH: Use arrow table to transfer dataset meta #663

Closed codingl2k1 closed 1 year ago

codingl2k1 commented 1 year ago

What do these changes do?

As the issue https://github.com/xorbitsai/xorbits/issues/638 has been fixed, we can use an arrow table instead of a pandas dataframe to transfer metadata.

Related issue number

Fixes #xxxx

Check code requirements

codecov[bot] commented 1 year ago

Codecov Report

Merging #663 (694134b) into main (34f7bca) will decrease coverage by 17.09%. The diff coverage is 96.77%.

@@             Coverage Diff             @@
##             main     #663       +/-   ##
===========================================
- Coverage   93.61%   76.52%   -17.09%     
===========================================
  Files        1025     1025               
  Lines       79726    79735        +9     
  Branches    16535    16538        +3     
===========================================
- Hits        74634    61017    -13617     
- Misses       3415    16349    +12934     
- Partials     1677     2369      +692     
Flag Coverage Δ
unittests 76.41% <96.77%> (-17.10%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
...on/xorbits/datasets/backends/huggingface/export.py 96.11% <93.33%> (-0.07%) :arrow_down:
python/xorbits/_mars/utils.py 73.08% <100.00%> (-16.19%) :arrow_down:
python/xorbits/datasets/iterable_dataset.py 96.79% <100.00%> (+0.09%) :arrow_up:

... and 276 files with indirect coverage changes