search

christophergrant / databricks-opentelemetry

0 stars 2 forks source link

Expose Structured Streaming backlog metrics #2

Open christophergrant opened 5 months ago

christophergrant commented 5 months ago

Streaming sources in Spark emit backlog metrics that indicate how behind a stream is. These metrics are helpful in determining a stream's performance relative to its source.

Streaming sources such as Autoloader, Kinesis, Kafka, and Delta emit these metrics.

This issue is to track efforts related to exposing these as metrics that can be centralized using OpenTelemetry.

souvik-databricks commented 5 months ago

@christophergrant , @cmgrant-db You can give me this issue to work on. thanks.

christophergrant commented 5 months ago

@souvik-databricks, great! If we use streamingQueryListeners, let's make sure to use spark.sql.streaming.streamingQueryListeners to make setup automatic and require 0 user application changes.