Check and write howto run Pelorus exporters with OpenShift prometheus from user monitoring stack

[mpryc]

After checking with Operator Framework on the slack channel the conclusions is as follows:

• Currently OLM does not allow to have optional dependencies during operator installation / subscription

• Subscription spec data could be used to optionally enable/disable OLM dependencies, but there is no such thing Today

• We could drop the prometheus/grafana dependencies from our operator, this however would require pre-requisite to deploy such operators manually

• We could have alternatively two separate pelorus operators, one with one without dependencies, this however could create confusion for the users

With the above a helm based installation is the easiest to achieve. Second option would be in my opinion removing prometheus/grafana dependencies from OLM and adding helm logic to discover OpenShift API if it's available from community operators. If it's not then we would not create those instances and deploy only exporters. The responsibility of configuring prometheus/grafana would be then on a user who opted-out from it. This will make UX experience worst from what we have now for the users which are deploying entire pelorus stack.

[beelzetron]

+1 on allowing to opt-out from prometheus/grafana installation. Some people have their own prometheus infrastructure they wanto to leverage.