Support multiple instances of application monitoring stack to be used to collect and show metrics for different users on the same cluster

[wei-lee]

In Integreatly, we are planning to have 2 instances of the application monitoring stack, one for SRE teams to monitor all the services, and the other one for end-users to use. In this case, some of the namespaces may need to be monitored by both stacks.

For example, the user-sso namespace needs to be monitored by the SRE team to make sure it is up and running. But also end-users need to see some of the application-specific metrics (like total login, failed login etc) through the user-facing metrics stack.

However, it doesn't seem to be possible to achieve this goal with the current implementation. The problem is that only 1 label (monitoring-key) will be used by the whole stack to look up for resources. If we can make the name of the label configurable, then it should be possible ( we just need to specify each stack to look for resources using different labels).

@david-martin @pb82 FYI

[pb82]

@wei-lee One way to implement this would be to use matchExpressions instead of matchLabels. This allows us to have a selector that matches multiple label values like for example:

- matchExpressions:
    - {key: 'monitoring-key', operator: In, values: [middleware, user]}

The application monitoring operator could pass the matchExpression (or the accepted label values) down to the Grafana and Prometheus operators (which already can accept it instead of matchLabels).

[wei-lee]

@pb82 sounds like a good solution to me! 👍