[TASK] Make use of metrics to create alerts for the relayer

[shekohex]

Overview

After creating the Grafana dashboard and utilizing the relayer metrics for monitoring in #388, we should also make use of this by creating an alert system that would notify us (using some sort of push notifications) with the following fields:

1. Relayer's account balance (for each configured network)
2. Total gas fee spent (across every configured network / Bridge)
3. Total Fees earned (for each configured VAnchor)

Task Checklist

• [x] Create the Alert on grafana [See: https://grafana.com/docs/grafana/latest/alerting]
• [x] Configure a slack bot/channel to send these alerts
• [ ] Send the updates on interval of 1 day.
• [x] Send an urgent alert if the account balance is lower than X native token.
• [x] Test the alerting system.

[Nutomic]

To give an update for this, it seems pretty much impossible to create daily alerts in Grafana. Its not mentioned in docs at all, the only reference I could find is this old thread and the suggested solution isnt working in current Grafana versions. So I think for daily updates its better to view the dashboard.

[shekohex]

Fair. So the other points we can do this right? like the Send an urgent alert if the account balance is lower than X native token.

[Nutomic]

Yes I just opened the PR for that: https://github.com/webb-tools/relayer/pull/479