[FEA] [Audit] - [SPARK-36030] Support DS v2 metrics at writing path

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs

https://nvidia.github.io/spark-rapids

Apache License 2.0

785 stars 228 forks source link

[FEA] [Audit] - [SPARK-36030] Support DS v2 metrics at writing path #3225

Closed nartal1 closed 3 years ago

nartal1 commented 3 years ago

Is your feature request related to a problem? Please describe. Metrics was added in https://github.com/apache/spark/commit/86d1fb4698 . We need to update GpuFileFormatDataWriter.scala as we had copied the code from FileFormatDataWriter.scala earlier.

tgravescs commented 3 years ago

the regular file based datasourcev2 write path is disabled in Spark right now so doesn't apply to our code at this point, it only applies to tables and custom sources. I don't see that this is needed until they enable that and we support or we have other datasource v2 writers that we want to use it. If they changed something that broke compilation that will be obvious and we can fix, otherwise not needed.