Add visibility to cron failures in production

[mostlikelee]

Goal

User story
As a Fleet engineer,
I want to be alerted when failures occur in Fleet server cron jobs
so that I can be proactive in resolving customer issues before they are reported.

Context

• Requestor(s): @mostlikelee
• Product designer: _____

Cron jobs tend to fail silently for different reasons in different environments with varying impacts. The following issues are examples of recent customer impacting cron failures where we were not alerted: https://github.com/fleetdm/fleet/issues/22366 https://github.com/fleetdm/fleet/issues/22364 https://github.com/fleetdm/fleet/issues/21292

Ideally we would like to have visibility into failures on cloud hosted AND self hosted environments.

Changes

Product

• [ ] UI changes: TODO
• [ ] CLI (fleetctl) usage changes: TODO
• [ ] YAML changes: TODO
• [ ] REST API changes: TODO
• [ ] Fleet's agent (fleetd) changes: TODO
• [ ] Activity changes: TODO
• [ ] Permissions changes: TODO
• [ ] Changes to paid features or tiers: TODO
• [ ] Other reference documentation changes: TODO
• [ ] Once shipped, requester has been notified

Engineering

• [ ] Feature guide changes: TODO
• [ ] Database schema migrations: TODO
• [ ] Load testing: TODO

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: TODO
• Risk level: Low / High TODO
• Risk description: TODO

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.

[lukeheath]

@mostlikelee Thanks for filing this. I am prioritizing to the drafting board for estimation and assigning to @sharon-fdm.

Where are you thinking we report failures to? #help-p2?

This is definitely something we need visibility into.

[mostlikelee]

help-p2 seems reasonable or possibly through datadog metrics, or a combination of both.