apache / celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

https://celeborn.apache.org/

Apache License 2.0

862 stars 351 forks source link

[CELEBORN-912][FOLLOWUP] Support columnar shuffle for Spark 3.5 #2710

Closed SteNicholas closed 1 week ago

SteNicholas commented 2 weeks ago

What changes were proposed in this pull request?

Introduce spark-3.5-columnar-shuffle module to support columnar shuffle for Spark 3.5.

Why are the changes needed?

1850 does not support columnar shuffle for Spark 3.5, in which version building `spark-3-columnar-shuffle` module has compilation error. The compilation error is caused by https://github.com/apache/spark/pull/40784, which incompatible changes move `InternalType` from `AtomicType` to `PhysicalDataType`.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI.

SteNicholas commented 2 weeks ago

Ping @kerwin-zk, @mridulm, @zhouyifan279. PTAL.