search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.78k stars 561 forks source link

streaming: back pressure for rearranged chain #5790

Open BugenZhao opened 1 year ago

BugenZhao commented 1 year ago

The rearranged chain in #1124 solves the issue of ingesting large historical data, and introduces a delicate way of deciding when to catch up with the upstream to finish the creating procedure. However, this relies on the assumption that the throughput of downstream MV is larger than the upstream, or it'll be a live lock.

The core reason is that the Chain bypasses the natural back pressure with active polling the upstream while rearranging the barrier to the downstream. So one solution is to manually introduce a token mechanism in the Chain to simulate a back pressure.

Here's a workable but not that elegant attempt -> #3899

github-actions[bot] commented 1 year ago

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.