Enhancement: Configure reapers to distribute deletions evenly instead of in periodic bursts

dynamic-entropy commented 4 months ago

Enhancement Description

Sites with slow deletions struggle during high loads, often exceeding storage capacity and failing to catch up. We need to optimize reaper configuration and scheduling to address this issue. This enhancement should consider factors such as site-specific deletion latency and deletion backlog.

In addition, auto-scaling and deploying reapers for specific sites during emergency high loads would be highly beneficial.

Use Case

During peak usage times, a site with slow deletions risks exceeding its storage capacity. Optimized reaper configurations and work scheduling will help manage deletions more efficiently.

Automatic scaling and deployment of reapers will ensure that deletion processes keep pace with increased writes. This will also eliminate the manual workload for operators.

Possible Solution

No response

Related Issues

No response

labkode commented 3 months ago

ericvaandering commented 1 month ago

This has been put as high with no plan how to do it as far as I can tell.

We have, in the past, stood up special reapers for things and we run a dedicated one for Tier0.

Can we talk about how to include this in the project in a realistic way?

dmwm / CMSRucio