Alert when jobs fail

[Razzmatazzz]

Working on troubleshooting exactly what causes jobs to fail to start running.

So far, have determined that jobs fail to start because the job is already running. I believe there are two likely culprits for this.

First, it's possible queries to the DB are stalling out and the job waiting for those queries therefore finishes. Currently, I've mocked up and deployed a timeout mechanism to guard against this.

Second, it's possible that the Cloudflare KV put calls are stalling. There is currently no timeout set for them. if the DB timeout fix doesn't work, this will be tried next. Upon further reflection, I think this is the more likely culprit. We occasionally used to get job error warnings about the put requests failing, but we no longer get those. For mysterious Cloudflare reasons, I would guess the requests just remain pending forever rather than throwing errors.

[github-actions[bot]]

👋 Thanks for opening a pull request!

If you are new, please check out the trimmed down summary of our deployment process below:

1. 👀 Observe the CI jobs and tests to ensure they are passing
2. ✔️ Obtain an approval/review on this pull request

3. 🚀 Branch deploy your pull request to production

   Comment .deploy on this pull request to trigger a deploy. If anything goes wrong, rollback with .deploy main

4. 🎉 Merge!

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy

[github-actions[bot]]

Deployment Triggered 🚀

Razzmatazzz, started a branch deployment to production

You can watch the progress here 🔗

Branch: alert-job-failed

[github-actions[bot]]

Deployment Results ✅

Razzmatazzz successfully deployed branch alert-job-failed to production

[Razzmatazzz]

.deploy