search

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
651 stars 119 forks source link

Fix CometShuffleManager to adapt IndexShuffleBlockResolver signature change #609

Closed kazuyukitanimura closed 1 day ago

kazuyukitanimura commented 4 days ago

Describe the bug

The head of Spark 3.4 branch changed the signature of IndexShuffleBlockResolver constructor compared to v3.4.1 In particular val taskIdMapsForShuffle: JMap[Int, OpenHashSet[Long]] = Collections.emptyMap() is added as 3rd argument that creates binary compatibility issue.

Steps to reproduce

Use latest snapshot of Spark 3.4

Expected behavior

No response

Additional context

No response

kazuyukitanimura commented 3 days ago

If this one merged https://github.com/apache/spark/pull/47148, we can close this one

viirya commented 3 days ago

That Spark PR is targeting the master branch, how does it do with Spark 3.4?

In any way, I think it is only an issue once you use Spark 3.4 with that signature change and Comet built with older Spark 3.4 without the change.