Setup alerts to monitor gatekeeper configuration

Background

OPA Gatekeeper is one of the critical component in the system and it is important to monitor gatekeeper availability, constraints and mutations count. For any reason, one of the mutations or constraints are deleted, the application workload may-not be scheduled.

Approach

Alert if gatekeeper pods is not available/ or not in running state
Expose gatekeeper metrics using Pod Monitor for prometheus to scrape
Setup alerts to monitor change in constraints, mutations and last run time of both. Some good example of metrics to monitor is mentioned in here: https://github.com/open-policy-agent/gatekeeper/issues/157#issuecomment-553015292

Which part of the user docs does this impact

https://github.com/ministryofjustice/cloud-platform-terraform-gatekeeper

Communicate changes

[ ] post for #cloud-platform-update
[ ] Weeknotes item
[ ] Show the Thing/P&A All Hands/User CoP
[ ] Announcements channel

Questions / Assumptions

Definition of done

[ ] readme has been updated
[ ] user docs have been updated
[ ] another team member has reviewed
[ ] smoke tests are green
[ ] prepare demo for the team

Reference

How to write good user stories

ministryofjustice / cloud-platform

Setup alerts to monitor gatekeeper configuration #4988