search

momentum-mod / website

Momentum Mod's main website.
https://momentum-mod.org
MIT License
58 stars 61 forks source link

Optimize replay file deletion #779

Closed tsa96 closed 3 months ago

tsa96 commented 1 year ago

We'll be deleting a lot of replay files from S3, when a user PBs, and when maps are updated (changing state during map review, patching a map to fix a catastrophic bug). In the latter case we'd be reaaaaally hammering S3/R2 if we do individual deletes.

Also, we're going to be adding a status flag to each run in the future, likely PENDING_VALIDATION, APPROVED. Can easily add another, PENDING_DELETION. When a run is deleted, we just update the service to PENDING_DELETION.

Then, have a new service that uses NestJS's @Cron decorator from the scheduler that periodically, within an interactive transaction gets all runs marked as PENDING_DELETION, creates a S3 delete multiple call, and assuming that's successful, actually deletes all those runs from DB, interactive transaction ends.

tsa96 commented 3 months ago

Blocking for now; with ongoing replay work it's possible the size of replays end up decreasing to a point where we don't need to delete

Gocnak commented 3 months ago

Deleting in R2 is completely free: https://developers.cloudflare.com/r2/pricing/#free-operations

tsa96 commented 3 months ago

Oh I hadn't realised. Closing in that case, only case I'm concerned about multi-deletes will be when doing whole leaderboard deletions https://github.com/momentum-mod/website/issues/916, can tackle that there.