search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.07k stars 582 forks source link

Avoid relying on barrier_interval_ms to increase operator cache eviction watermark #19403

Closed hzxa21 closed 4 days ago

hzxa21 commented 1 week ago

After #16087, we have switched to using sequence-based memory eviction policy instead of an epoch-based policy. This means we no longer need to rely on barrier to increase the watermark. However, it seems that we are still use barrier_interval_ms as the ticker interval here. This may result in delay in cache eviction when barrier_interval_ms is increased.

yuhao-su commented 1 week ago

This is a long existing issue. We should first make sure all executors are not evicting cache on barrier basis.

hzxa21 commented 1 week ago

This is a long existing issue. We should first make sure all executors are not evicting cache on barrier basis.

I think we already did? Just checked hash join and hash agg.

MrCroxx commented 5 days ago

This is a long existing issue. We should first make sure all executors are not evicting cache on barrier basis.

It had been done before the sequence-based operator cache eviction.