[OBSERVABILITY][SENTINEL] Setup Sentinel balance alerts based on Prometheus metrics

[philipandersson]

Acceptance criteria

• [ ] Establish balance thresholds for each network
• [ ] Use a alert-mananger to route alets to #notif-solvency-mainnets when balance falls under threshold

[philipandersson]

@d10r we would need your input on balance threshold here, thanks

[d10r]

I think we should avoid having to provide thresholds denominated in native coins. If we instead denominate them in "gas amount" or "nr of simple txs" (a simple tx being 21000 gas), with a bit of added logic the native coin amount can be auto-derived. Example implementation of such a logic: https://github.com/d10r/fund-account/blob/master/fund-accounts.js

The mechanism for adding/changing thresholds should be straightforward and not require assistance of an ops guy. E.g. a web UI, or change a json file in a git repo, or interact with a chatbot in slack.

[philipandersson]

@mmd-afegbua Is it doable to denominate in gas token?

This would be configurable in the alertsmanager config.