Critical alerts/network health monitoring

[pigmej]

As briefly discussed in Gdansk we need extend our network monitoring to have:

• sensible set of alerts
• sensible set of "network seems healthy"

To be able to achieve that we need to know and understand what is still considered good enough and what not.

See https://github.com/spacemeshos/Infrastructure/issues/56

Target date to finish initial work (list of metrics to collect, status of each, understanding existing infrastructure): 2023/06/26 Target date to finish collecting missing genesis-critical metrics, setting up dashboards, etc.: 2023/07/10

[lrettig]

The list is basically done, @noamnelke and I spent a lot of time on it over the last few days. We'll investigate what's already available and what's feasible to add before genesis.

[lrettig]

List of metrics to monitor is done. I'm in the process of investigating which metrics are already being collected and how they're being collected and monitored, and figuring out how to add stuff that's still missing.

[lrettig]

Mostly finished this investigation. Handed off to @pigmej.