apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.22k stars 437 forks source link

[GLUTEN-7126][VL][1.2] Port Fix shuffle spill triggered by evicting buffers during stop (apache#6422) #7991

Open kecookier opened 2 days ago

kecookier commented 2 days ago

… stop (#6422)

In VeloxHashBasedShuffleWriter::stop, evict partition buffers can allocating extra memory and triggers spill. In this case the target partition buffer should not get shrunk.

Added UT for this case.

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

(Fixes: #ISSUE-ID)

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

github-actions[bot] commented 2 days ago

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

marin-ma commented 1 day ago

cc: @weiting-chen

github-actions[bot] commented 18 hours ago

https://github.com/apache/incubator-gluten/issues/7126