search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.86k stars 569 forks source link

perf(stream): stateless 2-phase agg, state local hash agg #16179

Open st1page opened 5 months ago

st1page commented 5 months ago

We have had a stateless 2-phase simple agg to prevent single-point bottlenecks in simple aggregation with less costs. https://github.com/risingwavelabs/risingwave/blob/4292d9fdd53c776033013a2e088122b383ec980e/src/frontend/src/optimizer/plan_node/logical_agg.rs#L196-L206 https://github.com/risingwavelabs/risingwave/blob/cd28ff0e23f5cb97b2b7b2d3ad3f0349d6379789/src/stream/src/executor/stateless_simple_agg.rs#L29

But we now found when a hotspot key exists. that is found in NEXMark q15. https://risingwave-labs.slack.com/archives/C0543LNH0KU/p1699324490070239?thread_ts=1699323766.819059&cid=C0543LNH0KU. The stateless 2-phase optmization is also needed.

stdrc commented 5 months ago

Any more information?

st1page commented 5 months ago

Any more information?

Yes I will add it.

github-actions[bot] commented 2 months ago

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean. Don't worry if you think the issue is still valuable to continue in the future. It's searchable and can be reopened when it's time. 😄