search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.78k stars 561 forks source link

feat(compaction): Support full compaction at table granularity #14209

Open Li0k opened 8 months ago

Li0k commented 8 months ago

In hummock, automatically triggered compaction and manual compaction are supported to meet the needs of most scenarios. However, each time manual compaction is initiated, only one compact task is triggered, which cannot meet the needs of full compaction.

In what scenarios does Full Compaction need to be introduced?

  1. When there is a large amount of redundant data in hummock, it needs to be cleaned.
  2. When a table_id contains a large number of stale keys, it needs to be cleaned immediately to improve query performance.
  3. When testing, it is necessary to verify whether the newly introduced compaction cleaning logic can work.
  4. When under low load, we can trigger full compaction to reorganize the data in hummock.

feat request:

  1. trigger full compaction at table granularity via SQL request by @chenzl25
  2. trigger full compaction to verify the table-watermark cleaning capabilities request by @wenym1
github-actions[bot] commented 6 months ago

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.