search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.98k stars 575 forks source link

Provide a way to check backfilling rate (in rows/s). #19120

Open neverchanje opened 2 days ago

neverchanje commented 2 days ago

We received a complaint from customer about the lack of the backfilling rate. So far, we've planned to show "elapsed time" and "progress in percentage" on ui, but the backfilling rate is an essential metric to include as well.

Please comment within the issue about what's the approach I can use to obtain this data. cc @xxchan

xxchan commented 2 days ago

On Prometheus/Grafana, we have:

With SQL SHOW JOBS or select * from rw_ddl_progress, there's a progress string, can be like:

So there's no direct access to backfilled rows. If needed, we may add a column besides the progress string, as https://github.com/risingwavelabs/risingwave/pull/18112#discussion_r1741571151.