Improve and increase coverage of alerting about issues in our system

[arealmaas]

Videreføring av issue: https://github.com/digdir/dialogporten/issues/75

Following up on the mentioned issue, we need to improve alerts and insight into errors/issues in our system.

Considerations:

• Errors and outages in (Maskinporten, Autorisasjon, Events)
  • Health check that verifies that the well-known endpoint is reachable.

Her er en liste over typiske feilsituasjoner som vi trenger å varsle om:

• Degradation/bortfall av komponenter (liveness/readiness)
• Feil fra/bortfall eksterne dependencies (Maskinporten, Autorisasjon, Events)

Andre situasjoner som indikerer hendelser som kan negativt påvirke tjenestenivå:

• Spikes i antall 4xx-feil
• Spikes i trafikk fra enkelt-parter

Infrastructure:

• Postgresql (slow queries, failed queries, high cpu/mem, database size)
• Redis (slowness, high cpu/mem)
• Servicebus (dead-letter queue, slow consumption time, cpu/mem)
• Container apps (restart count, crashlooping containers, high cpu/mem)

Create Slack-channels for each environment? To make clear the severity of alerts

- [ ] Create Slack-channels for each environment? To make clear the severity of alerts

[arealmaas]

Discussions:

https://digdir.slack.com/archives/C079D6PAGDS/p1728552564466459 https://digdir.slack.com/archives/C079XRW5G5A/p1728559035522959

[arealmaas]

Alerting about issues related to metrics would be possible to do in the grafana-dashboard if we decide to go for that. I suggest waiting with this task until we look into https://github.com/digdir/dialogporten/issues/1456