risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.96k stars 574 forks source link

perf: optimize `StateTable::write_chunk` #11377

Open kwannoel opened 1 year ago

kwannoel commented 1 year ago
  1. We should check vis before serializing chunks, so we can use the right iteration (use iterate rows / use rows_with_holes / consider serializing columns).
  2. Value chunk should be treated the same way as key chunks. If we skip key datum serialization because of viz, we should do so for values as well.
  3. After serializing chunks, we can immediately insert them, rather than check vis again.
  4. Ensure vnode_and_pks and the subsequent code is statically branched on USE_WATERMARK_CACHE.

We need better benchmarks for write_chunk:

kwannoel commented 1 year ago

So far PR #11455 shows that there is mix of improvement and regression, hence I'm hesistant to merge it. Welcome any comments.

github-actions[bot] commented 4 months ago

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.