search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.22k stars 437 forks source link

[GLUTEN-7750][VL]: store unsafe batches data #7944

Open Zand100 opened 1 week ago

Zand100 commented 1 week ago

What changes were proposed in this pull request?

Adds a binary container implementing a Spark MemoryConsumer to be used instead of Array[Array[Bytes]]. Uses that binary container with the existing object handles in ColumnarBuildSideRelation.

(Fixes: https://github.com/apache/incubator-gluten/issues/7750)

Duplicate of https://github.com/apache/incubator-gluten/pull/7902

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

github-actions[bot] commented 1 week ago

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

github-actions[bot] commented 1 week ago

https://github.com/apache/incubator-gluten/issues/7750