Open ringerc opened 1 month ago
I feel that it's a request which would be better addressed by the ScrapeConfig CRD. Also updating the RBAC permissions of the Prometheus service account isn't something under the responsibillity of the operator right now and I wouldn't change this.
Component(s)
ServiceMonitor, PodMonitor
What is missing? Please describe.
A way to configure prometheus and the operator to discover pods and services to scrape only in namespaces matching a label selector. In a way that means Prometheus won't try to enumerate workloads in non-matching namespaces where it may lack RBAC to list them.
PodMonitor
andServiceMonitor
'sspec.namespaceSelector
is a list of namespaces rather than an actual selector. It cannot match the namespaces to search by label.Where Prometheus instance deployed into an environment with strong RBAC (like a default Openshift cluster) it will not have permission to enumerate all pods in all namespaces. It can be given per-namespace permissions by deploying a suitable
Role
andRoleBinding
into the namespace(s). But currently there is no way to tell it to only look for resources in namespaces satisfying a label. If the namespace(s) containing the workloads targeted by thePodMonitor
orServiceMonitor
are not known in advance, this means there's no way to deploy a singlePodMonitor
orServiceMonitor
to scrape them.The
Prometheus
object'sspec.podMonitorNamespaceSelector
can be used to tell the operator to look forPodMonitor
s andServiceMonitor
s in namespaces matching a label selector. But in this case thePodMonitor
orServiceMonitor
(s) must be duplicated into each namespace.Instead, it would be helpful to be able to set a label selector for the namespaces searched for workloads by a
PodMonitor
orServiceMonitor
, so the monitor can discover workloads only in namespaces it's instructed to search. Ideally theprometheus-operator
could also assist with Prometheus RBAC, by injecting theRole
andRoleBinding
to allow Prometheus to enumerate scrape-able workloads in suitably annotated namespaces.Ideally something like this (nonexistent) config:
and in
monitoring
a role and rolebinding is also required
Describe alternatives you've considered.
Use per-namespace
PodMonitor
s, but:To do it:
Copy the
PodMonitor
s andServiceMonitor
s to each namespace along with theRole
andRoleBinding
needed to permit Prometheus to discover target workloads. Add a label to the namespaces to indicate that they're enabled for prometheus monitoring. Leave the pod and service monitors'.namespaceSelector
blank so only the current namespace is checked.Use the
Prometheus
object'sspec.podMonitorNamespaceSelector
andspec.serviceMonitorNamespaceSelector
to match namespaces that have the needed role and rolebinding.e.g.:
This works, but requires copying a lot of resources around, and it creates new targets per namespace.
Environment Information.
Environment
Kubernetes Version: v1.29.2 (kind), v1.28.10 (openshift) Prometheus-Operator Version: v0.75.1