search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
823 stars 236 forks source link

Execute `from_json` with struct schema using `JSONUtils.fromJSONToStructs` #11618

Closed ttnghia closed 2 days ago

ttnghia commented 1 month ago

This adopts the newly implemented JNI function JSONUtils.fromJSONToStructs() to parse the input strings columns into a structs column, which is the case of calling from_json SQL function with struct schema. By replacing the Scala code entirely by native code, we can avoid a lot of overhead and optimize runtime performance.

Closes https://github.com/NVIDIA/spark-rapids/issues/11560.

This will also close the following issues:

Depends on:

ttnghia commented 2 days ago

build