risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.03k stars 577 forks source link

nightly-20240913 sysbench MYSQL & PG CDC consume lag performance degradation #18572

Open cyliu0 opened 1 month ago

cyliu0 commented 1 month ago

Describe the bug

http://metabase.risingwave-cloud.xyz/dashboard/985-sysbench-pg-cdc-100-tables-1m-rows-medium-arm-all-affinity-history?start_date=2024-08-07

http://metabase.risingwave-cloud.xyz/question/32124-sysbench-pg-cdc-100-tables-1m-rows-medium-arm-all-affinity-avg-cdc-consume-lag-ms-history-tltb-2960?start_date=2024-04-01

image

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20240913

Additional context

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240913

StrikeW commented 1 month ago

The avg source throughput doesn't drop. So the overall performance is fine. May need to take a deep look of the consume lag metric.

image