search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
785 stars 228 forks source link

[Audit] Verify GpuCreateNamedStruct works with updated SQL syntax in Spark 3.4 #6749

Open NVnavkumar opened 1 year ago

NVnavkumar commented 1 year ago

Describe the bug Spark 3.4 updated how field names are preserved and ensured dot-notation and index-notation are equivalent. We should verify that this behavior works in Spark 3.4 with the RAPIDS Accelerator.

Spark commit - https://github.com/apache/spark/commit/e5914e199c

revans2 commented 1 year ago

From looking at the patch and the code for GPuCratedNamedStruct I don't think we will be impacted. The name of each field is passed into the constructor. But this is just to add a test to verify it, and that should be relatively simple to do.