Alerts for stuck messages that should’ve been deliverable

[daniel-savu]

We have no way of alerting based on messages that should be deliverable but are stuck (e.g. those who've had their IGP allowance depleted). Concrete action items to consider:

• expose an igp_depleted: bool label for each message, that would help us detect most unnecessarily stuck messages (where the relayer shouldn't have sent a txs that reverted); @tkporter suggested having DepletedIgpAllowance be a new status label (in https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4603#issuecomment-2388422855). But there could be other reasons why messages become stuck.
• capture the history of statuses a prep queue message went through, to allow for more expressive alerting (including cases of IGP allowance depletion). However storing this in metric seems tricky to me, because the label could take on an unbounded amount of values (i.e. all status labels transition histories)

https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/4602 could be considered a complement to this task, since it would greatly improve the debugging process

[tkporter]

Copying a comment from https://discord.com/channels/935678348330434570/1290349466578190358

I wonder if we should have a special retry reason for a message that has an expenditure so we can spot this in the future
I think it's probably good to have the expenditure logic to protect ourselves but it's rare enough where it should probably be a low urgency alert / something to probe into?
like feels a little bad that we got reports of this from others instead of spotting ourselves