Do not hesitate talking to devops for help in using Grafana and Alert Manager.
DoD
define some metrics we can build alerts on (e.g. number of 4XX-5XX responses, requests duration)
build some new charts for those metrics (you can copy the query definition from the nginx dashboards and tweak it a little, we cannot directly use the existing charts because alerts cannot include "template variables" (the four dropdowns you see at the top of the screen: Namespace, Controller Class, Controller, Ingress))
create alerts from these new charts, use slack as a notification channel
We do have a nginx ingress dashboard here: https://grafana.emeris.allinbits.com/d/nginx/nginx-ingress-controller?orgId=1
Do not hesitate talking to devops for help in using Grafana and Alert Manager.
DoD