Don't create cluster-wide resources if possible

[patricklucas]

The Prometheus and Grafana Helm charts both create some cluster-level resources, causing conflicts if they are installed multiple times in the same cluster.

We might be able to improve that here by tweaking their values files and possibly creating some resources manually.

Namely, these are the ones I've found:

• clusterroles/prometheus-server
• clusterrolebindings/prometheus-server
• clusterroles/grafana-clusterrrole
• clusterrolebindings/grafana-clusterrrolebinding
• podsecuritypolicies/grafana
• podsecuritypolicies/grafana-test

For Prometheus we should be able to set rbac.create=false, but I think we'll need to create a rolebinding ourselves so it can discover pods in the same namespace. (create rolebinding --clusterrole admin --serviceaccount prometheus-server should do it).

For Grafana, rbac.pspEnabled=false and rbac.namespaced=true might get us where we want.

[shalberd]

Good point. I wouldn't even use those instances of Prometheus and Grafana in the helm chart and instead use, at least on Openshift 4.x, what comes with the cluster, namely monitoring for user defined projects for all non-kube and non-openshift namespaces. https://docs.openshift.com/container-platform/4.11/monitoring/enabling-monitoring-for-user-defined-projects.html

Grafana in the namespace possibly via Grafana Operator https://github.com/grafana-operator/grafana-operator and best installed via Operator Lifecycle Manager (OLM). Or go with Mimir instead of Grafana Operator: https://grafana.com/docs/mimir/latest/operators-guide/run-production-environment-with-helm/