Prevent scheduler failures

andrew-nowak commented 7 months ago

What does this change?

If the Runnable in a scheduler throws, the scheduler will stop and no longer run, even if future executions would succeed.

eg. If a quota report fails to parse, sending through a fixed report will still leave no quotas available until media-api is redeployed/restarted.

I checked the codebase for other schedulers and spotted that Reaper would be susceptible to the same problem, so put a try/catch in there too.

How should a reviewer test this change?

A bit hard to trigger an error like this. Possibly we can drop a malformed report into the mail bucket locally or on TEST? give me a shout if you'd like to pair on trying it.

How can success be measured?

Intermittent failures do not cause consistent failure 👍

github-actions[bot] commented 7 months ago

Deploy build 12168 to TEST

All deployment options

- [Deploy build 12168 to TEST](https://riffraff.gutools.co.uk/deployment/deployAgain?project=media-service%3A%3Agrid%3A%3Aall&build=12168&stage=TEST&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 12168 to TEST by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=media-service%3A%3Agrid%3A%3Aall&build=12168&stage=TEST&updateStrategy=MostlyHarmless)

From guardian/actions-riff-raff.

prout-bot commented 7 months ago

Seen on auth, metadata-editor, leases, cropper, collections, media-api (merged by @andrew-nowak 8 minutes and 46 seconds ago) Please check your changes!

prout-bot commented 7 months ago

Seen on image-loader, thrall, usage (merged by @andrew-nowak 8 minutes and 52 seconds ago) Please check your changes!

prout-bot commented 7 months ago

Seen on kahuna (merged by @andrew-nowak 9 minutes and 46 seconds ago) Please check your changes!

guardian / grid