[Spark] Log duration of DeltaSource operations

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.18k stars 1.62k forks source link

[Spark] Log duration of DeltaSource operations #2846

Closed tomvanbussel closed 6 days ago

tomvanbussel commented 2 months ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

This PR adds logging the main DeltaSource operations to allow us to measure the duration of these operations.

How was this patch tested?

Existing tests to make sure nothing breaks.

delta-io / delta

[Spark] Log duration of DeltaSource operations #2846

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?