Establish monitoring and ops-alerts/metrics

olebhansen commented 2 months ago

Need to know:

Usual RED+S metrics (https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/)
Alarm on when the synch fails
Some sort of traffic-light aggregation to indicate that the service works (emitting metrics and no failures), yellow (no metrics) and red (emitting failures)

### Prerequistites:
- [ ] https://github.com/Altinn/altinn-profile/issues/196

[ ] we have a visual presentation of (number of failing sync jobs) / (time)
[ ] we have a visual presentation of (number of updated contacts) / (sync job)|
[ ] we have a visual presentation of the request response time
[ ] we have a visual presentation of the CPU usage of the application and the DB server
[ ] the existing Prometheus/Grafana setup for Profile, using the ingress route, has an SLO value that is more relevant to us (99.0->99.9%)

hansdragnes commented 1 month ago

As a start, consider devloping a POC, in order to build experience on "LGTM" (?) Create separate exploration task for the purpose.

hggutvik commented 4 weeks ago

Acceptance criterias defined 24. oct in a team meeting, after discussion

Altinn / altinn-profile