dmwm / CMSRucio

7 stars 31 forks source link

Enhancement: Configure reapers to distribute deletions evenly instead of in periodic bursts #829

Open dynamic-entropy opened 3 months ago

dynamic-entropy commented 3 months ago

Enhancement Description

Sites with slow deletions struggle during high loads, often exceeding storage capacity and failing to catch up. We need to optimize reaper configuration and scheduling to address this issue. This enhancement should consider factors such as site-specific deletion latency and deletion backlog.

Use Case

During peak usage times, a site with slow deletions risks exceeding its storage capacity. Optimized reaper configurations and work scheduling will help manage deletions more efficiently.

Possible Solution

No response

Related Issues

No response

labkode commented 2 months ago

Related to https://github.com/rucio/rucio/issues/6512

ericvaandering commented 1 week ago

This has been put as high with no plan how to do it as far as I can tell.

We have, in the past, stood up special reapers for things and we run a dedicated one for Tier0.

Can we talk about how to include this in the project in a realistic way?