Message lifecycle monitoring

[daniel-savu]

• it used to be impossible to know why messages are stuck in the prepare queue
• since then, we added an operation_status label to prep queue metrics that can be used in grafana. See this grafana graph for celo prep queues
  • the implementation has a bug where operation_status labels seem to not be persisted, in spite of previous work on this: https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4043
• based on the status labels, we also added a relayer endpoint (/list_operations) that prints the entire prep queue for a given remote: https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4089
  • the problem is that it doesn't display the message ID, gas payment status (amounts paid and expended so far), and it also doesn't show the history of statuses the message went through (e.g. FirstPrepareAttempt -> ReadyToSubmit -> Retry(RevertedOrReorged) -> GasPaymentRequirementNotMet, which would reveal that the relayer consumed the IGP allowance of the message on txs that reverted)

Concrete action items:

1. fix operation_status labels not being persisted
2. add tracking for historic statuses a PendingOperation went through
3. Add message ID, gas payment status (amounts paid and expended so far) and historic operation statuses to the list_operations endpoint

When adding details to the list_operations endpoint, follow the tips here: https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/4610#issuecomment-2389769026

[tkporter]

I feel like we could maybe get decently far here with better insight into the relayer's logs (like via https://github.com/hyperlane-xyz/issues/issues/1259)