Make Prometheus Operator find ServiceMonitors that are outside of monitoring namespace

[exArax]

Hi,

I am configuring a pod, a service and a monitorservice in a namespace and I cannot find in the Prometheus GUI neither on Service Discovery or Targets. Should I change some files in the the manifests files ? What should I do?

logs:

level=error ts=2022-02-08T14:30:33.415Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"accordion-ovr-0-0-3-123\"" level=error ts=2022-02-08T14:30:34.763Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"accordion-ovr-0-0-3-123\"" level=error ts=2022-02-08T14:30:49.647Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"accordion-ovr-0-0-3-123\""

[carlosedp]

You need to set the namespace config to allow Prometheus to find it like in https://github.com/carlosedp/cluster-monitoring/blob/514aa37f9a0d32d729ac676967febf0e97a61917/modules/metallb.jsonnet#L8-L10

[badnetmask]

This conversation just made me realize I do have metallb enabled on vars.jsonnet, the proper namespace set in metallb.jsonnet, but I don't see anything related to it being applied.

# kubectl apply -f cluster-monitoring/manifests
(...)
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created
deployment.apps/speedtest-exporter created
service/speedtest-exporter created
servicemonitor.monitoring.coreos.com/speedtest-exporter created
servicemonitor.monitoring.coreos.com/traefik created

[carlosedp]

For example, when metallb module is enabled, it applies the additional files:

?? manifests/metallbexporter-clusterRole.yaml
?? manifests/metallbexporter-clusterRoleBinding.yaml
?? manifests/metallbexporter-service.yaml
?? manifests/metallbexporter-serviceMonitor.yaml

They create the role and binding allowing Prometheus to monitor the application in a different namespace.

[exArax]

You need to set the namespace config to allow Prometheus to find it like in

https://github.com/carlosedp/cluster-monitoring/blob/514aa37f9a0d32d729ac676967febf0e97a61917/modules/metallb.jsonnet#L8-L10

Hi Carlos, I made it work. I have changed somethings in the prometheus-roleSpecificNames.yaml file and now it is fine. In the related ServiceMonitor I have added the namespace selector with the value any. Is this different from what you suggested ?

[carlosedp]

I don't think "any" would work. You need to set the correct namespace.

[exArax]

I don't think "any" would work. You need to set the correct namespace.

Actually it did work. I created a ServiceMonitor under the monitoring namespace which can find Services that are on different namespaces. So I did a workaround from what I was describing before.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: monitorable-vm-windows-exporter
  namespace: monitoring
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      type: monitorable-vm-windows-exporter
  namespaceSelector:
    any: true
  endpoints:
    - port: metrics
      interval: 60s

[carlosedp]

Believe this can be closed now.