search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.8k stars 564 forks source link

`meta.compaction_config` can affect s3 request rates. #14725

Open kwannoel opened 8 months ago

kwannoel commented 8 months ago

Can use this to reproduce: https://github.com/risingwavelabs/risingwave/pull/14722.

When this is present:

[meta.compaction_config]
level0_tier_compact_file_number = 6
level0_overlapping_sub_level_compact_level_count = 3
level0_max_compact_file_number = 96

The tests will fail with SlowDown error from s3 (minio).

When it's not present, the tests will complete really quickly.

What's the reason this can affect the test performance?

hzxa21 commented 8 months ago

I think it is because #13331 implicitly bumps the checkpoint frequency from ckpt every 2 barriers to ckpt every 1 barrier. This has several implications:

  1. CN's write to S3 rate is doubled
  2. Storage L0 files increasing rate is doubled
  3. With the compaction config you have pasted, compaction is triggered more aggressively because of 2.

All these can lead to an increase in S3 request rate. @Li0k has merged #14601 to modify the default compaction config accordingly. I think we missed to update the ci config, which is set explicitly rather than using default.

github-actions[bot] commented 3 months ago

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.