search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.78k stars 561 forks source link

mem table spill only uses the last 12 bits of epoch #14892

Open wcy-fdu opened 7 months ago

wcy-fdu commented 7 months ago

This is a feature request by recursive cte, which tend to use the high 4 bits of the last 16 bits of epoch to iterate.

After some offline discussion, this feature will not change the semantics of the current operator reading storage, and mem table spill only uses the last 12 bits of epoch (spill offset <= 4096) can be acceptable after Global memory statistics and dynamic flush threshold for mem table is implemented. We only need to change the write logic when building SSTs, limiting the spill offset up to 4096 times when writing.

Let's start this change after global memory statistics and dynamic threshold is implemented.

github-actions[bot] commented 1 month ago

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean. Don't worry if you think the issue is still valuable to continue in the future. It's searchable and can be reopened when it's time. 😄