xorbitsai / xorbits

Scalable Python DS & ML, in an API compatible & lightning fast way.
https://xorbits.io
Apache License 2.0
1.06k stars 67 forks source link

CHORE: Fix `PerformanceWarning` in `pivot` combine stage #747

Open ChengjieLi28 opened 8 months ago

ChengjieLi28 commented 8 months ago

What do these changes do?

Fix this warning in pivot op combine stage.

performancewarning: dataframe is highly fragmented. this is usually the result of calling `frame.insert` many times, which has poor performance. consider joining all columns at once using pd.concat(axis=1) instead. to get a de-fragmented frame, use `newframe = frame.copy()`

Related issue number

Fixes #xxxx

Check code requirements

codecov[bot] commented 8 months ago

Codecov Report

Merging #747 (f1587de) into main (b320ca3) will decrease coverage by 27.67%. The diff coverage is 0.00%.

@@             Coverage Diff             @@
##             main     #747       +/-   ##
===========================================
- Coverage   77.29%   49.62%   -27.67%     
===========================================
  Files        1058     1058               
  Lines       79780    79779        -1     
  Branches    16504    12433     -4071     
===========================================
- Hits        61663    39593    -22070     
- Misses      15724    38595    +22871     
+ Partials     2393     1591      -802     
Flag Coverage Δ
unittests 49.61% <0.00%> (-27.56%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
python/xorbits/_mars/dataframe/base/pivot.py 23.70% <0.00%> (+0.17%) :arrow_up:

... and 639 files with indirect coverage changes