search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
822 stars 235 forks source link

Update to Spark 4.0 changing signature of SupportsV1Write.writeWithV1 #11739

Closed jlowe closed 23 hours ago

jlowe commented 1 day ago

After SPARK-50315 SupportsWriteWithV1.writeWithV1 no longer returns a value, causing a build error for Spark4.0. This method simply returned Nil before, so rather than shim this for Spark 4 it updates the code to return Nil directly.

Fixes #11741

jlowe commented 1 day ago

build

revans2 commented 1 day ago

Looks like the build now works, but a number of generated files are out of date for spark 4.0. Do we want to generate those here? or would a separate PR be better for that?

jlowe commented 1 day ago

I went ahead and updated the tools support files for Spark 4.0, since the point of this PR is to fix builds for Spark 4.0 including the premerge build checks.

jlowe commented 1 day ago

build