Open krishmanoh2 opened 3 years ago
This will likely be done opportunistically within the next couple of months, after the exactly once and idempotent ingest tasks wrap up. Currently viewing it as low priority, but will revisit if we have a fuller understanding of customer need (specifically how customers are currently monitoring consumer lag and why that paired with the existing dataflow lag metrics are not sufficient).
Customers are relying on visual cues - materialize prometheus metrics in grafana to calculate lag.
Few notes from recent discussions: mz has a statistics_interval_ms
that can be set on kafka source creation to dump detailed statistics into the materialized.log. That's been broken for awhile, but is now working on the latest main branch. The likely path forward is to see if those metrics can be recorded in a system table instead of just written to the log file - potential complications are around the overhead of that stats collection since it incurs some synchronization, a number of hdrhistograms, and json ser/deser.
It is hard to calculate source lag with sql on mz today.
For kafka as source, if we can show the latest available kafka offset (committed/uncommitted), would help customers/prospects calculate lag. Ideally, we would also show throughput and latency in this view so the customer can determine how long it will take to catch up (if there is a lag).