[SPARK] Add SQL metrics for ConvertToDelta

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.62k stars 1.71k forks source link

[SPARK] Add SQL metrics for ConvertToDelta #3841

Closed olaky closed 6 days ago

olaky commented 2 weeks ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

Add the metric "numConvertedFiles" to the ConvertToDelta command node

How was this patch tested?

Added a new test validating the values

Does this PR introduce any user-facing changes?

The metric become visible in the plan (i.e. Spark UI)