Improve and increase coverage of alerting about issues in our system - Githubissues

digdir / dialogporten

Dialogporten - common API and and metadata state store for digital dialogs

https://docs.altinn.studio/dialogporten

MIT License

1 stars 3 forks source link

Improve and increase coverage of alerting about issues in our system #1261

Open arealmaas opened 1 week ago

arealmaas commented 1 week ago

Videreføring av issue: https://github.com/digdir/dialogporten/issues/75

Following up on the mentioned issue, we need to improve alerts and insight into errors/issues in our system.

Considerations:

Errors and outages in (Maskinporten, Autorisasjon, Events)
- Health check that verifies that the well-known endpoint is reachable.

Her er en liste over typiske feilsituasjoner som vi trenger å varsle om:

Degradation/bortfall av komponenter (liveness/readiness)
Feil fra/bortfall eksterne dependencies (Maskinporten, Autorisasjon, Events)

Andre situasjoner som indikerer hendelser som kan negativt påvirke tjenestenivå:

Spikes i antall 4xx-feil
Spikes i trafikk fra enkelt-parter

Infrastructure:

Postgresql (slow queries, failed queries, high cpu/mem, database size)
Redis (slowness, high cpu/mem)
Servicebus (dead-letter queue, slow consumption time, cpu/mem)

Create Slack-channels for each environment? To make clear the severity of alerts

- [ ] Create Slack-channels for each environment? To make clear the severity of alerts

arealmaas commented 1 week ago

Discussions:

https://digdir.slack.com/archives/C079D6PAGDS/p1728552564466459 https://digdir.slack.com/archives/C079XRW5G5A/p1728559035522959