[reaper] reap at configured rate (rather than dynamic based on ingestion rate) - Githubissues

guardian / grid

The Guardian’s image management system

https://www.theguardian.com/info/developer-blog/2015/aug/12/open-sourcing-grid-image-service

Apache License 2.0

1.44k stars 119 forks source link

[reaper] reap at configured rate (rather than dynamic based on ingestion rate) #4169

Closed twrichards closed 9 months ago

twrichards commented 9 months ago

Thanks to #4168, we can see that although we do catch-up slightly each night, the amount of reapable images is slowing getting away from us...

I had become a little fixated on avoiding config and tried to be too clever (trying to base the reaping rate on the 7day rolling average of ingestion in #4145) which isn't keeping up and is harder to reason about.

Should've heeded the suggestion from prophet @andrew-nowak that reaping rate should be fixed/configured.

... this PR does exactly that; [scheduled] reaping is now based on reaper.countPerRun config property.

prout-bot commented 9 months ago

Seen on image-loader (merged by @twrichards 7 minutes and 47 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on collections, media-api (merged by @twrichards 7 minutes and 58 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on auth (merged by @twrichards 8 minutes and 4 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on kahuna (merged by @twrichards 8 minutes and 11 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on thrall, leases (merged by @twrichards 8 minutes and 19 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on thrall, leases (merged by @twrichards 8 minutes and 19 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on thrall, leases (merged by @twrichards 8 minutes and 19 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on metadata-editor, cropper (merged by @twrichards 8 minutes and 53 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on usage (merged by @twrichards 9 minutes and 5 seconds ago) Please check your changes!

twrichards commented 9 months ago

Just a lil update; you can see this has been very effective... You can see in PROD...

the red SoftReapable line begin, when #https://github.com/guardian/grid/pull/4168 was released (which shows it generally trending up, despite fluctuations)
the point this PR was released, the SoftReaped count jump up to 1000 per 15mins (up from the dynamic ~510)
the total SoftReapable count come down more aggressively since this PR
until it reaches sub-1000, then the SoftReaped count becomes more scattered as it soft deletes only the ones available 🎉

The hard reaping metrics will kick in later this week once the big backlog of millions of images reaches two weeks in soft deleted state.