search

numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io/
Apache License 2.0
1.05k stars 103 forks source link

Expose pipeline processing lag metrics #1821

Closed whynowy closed 1 month ago

whynowy commented 1 month ago

Summary

  1. Pipeline processing lag (max watermark - min watermark) - We have pipeline processing lag displayed on the UI, which is also needed to expose as a metric.
  2. Watermark compared with current time (Current time - max watermark)

It could be exposed from daemon service.

Screenshot 2024-07-12 at 11 32 39 AM

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

whynowy commented 1 month ago

@samhith-kakarla

whynowy commented 1 month ago

Closed by https://github.com/numaproj/numaflow/pull/1839