Alert status manager gets stuck with empty log ID

[mastercactapus]

Describe the Bug: If an alert's status is updated, without a corresponding entry in the alert_logs table for any reason, the status manager gets stuck with errors, potentially blocking status updates for other alerts, and the alert in question never has things like Slack messages update.

Steps to Reproduce: Since it's an edge case, the exact steps are uncertain. But it follows this pattern:

1. Update an alert
2. Delete the associated log entry before the status manager has processed the alert

Expected Behavior: On failure to record a log, an error should be logged, but status updates should continue to work even in the absence of a specific alert's log.

Observed Behavior: Status updates break entirely because they stay "stuck" on the error-state alert.

Screenshots/Stack Traces:

2024-03-22 10:37:36.852 time="2024-03-22T15:37:36Z" level=error AuthSystemComponent=Engine SQLErrDetails="Failing row contains (ffffffff-ffff-ffff-ffff-ffffffffffff, alert_status_update, null, 2024-03-22 15:37:36.840009+00, pending, 2024-03-22 15:37:36.840009+00, , null, null, 0, null, null, null, 78595, null, null, null, null, null, null, 0, eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee, null, null, null)." Trigger=INTERVAL error="Engine.StatusUpdateManager: send channel status update message: ERROR: new row for relation "outgoing_messages" violates check constraint "om_status_update_log_id" (SQLSTATE 23514)

Application Version: Discovered on GoAlert with GitCommit 5e8ce484c02987983668fbeeb4a11431f4e85389.

Desktop: Not applicable as it is a backend/engine issue.

Additional Context Fix notes:

• Update status manager to record a zero instead of null to the alert_log_id column in this case
• Update sendmessage.go to skip fetching the log if it's zero and use an empty string (also if the log lookup returns ErrNoRows)
• Ensure notification senders handle this case appropriately
• Add a smoke test illustrating this state and asserting the correct behavior (can likely copy existing status smoke test and modify for this bug)

[mastercactapus]

It might be helpful to emit an error upon hitting the missing log entry case (as it shouldn't happen) as long as it's in a way that the error will only be logged once.

[mastercactapus]

Start of fix: https://github.com/target/goalert/tree/fix-status-missing-log

• needs validation
• needs smoke test