search

prometheus-community / helm-charts

Prometheus community Helm charts
Apache License 2.0
5.08k stars 5.01k forks source link

[kube-prometheus-stack] namespace does not exists #3741

Open urbaman opened 1 year ago

urbaman commented 1 year ago

Describe the bug a clear and concise description of what the bug is.

Hi,

I'm trying to add the cert-manager namespace to the scoped namespaces, and added it in the helm chart:

  namespaces:
    releaseNamespace: true
    additional:
      - calico-apiserver
      - calico-system
      - cert-manager
      - cloudflared
      - default
      - gpu-feature-discovery
      - heimdall
      - homer
      - kube-node-lease
      - kube-public
      - kube-system
      - kubernetes-dashboard
      - kubescape
      - longhorn-system
      - metallb-system
      - monitoring
      - nfs-provisioning
      - node-feature-discovery
      - nvidia-device-plugin
      - portainer
      - teleport-agent
      - teleport-cluster
      - tigera-operator
      - traefik
      - traefik-external

But it does not seem to work properly:

msg="get namespace to enqueue Prometheus instances failed: namespace \"cert-manager\" does not exist"

And it does exists:

kubectl get namespaces
NAME                     STATUS   AGE
calico-apiserver         Active   27h
calico-system            Active   27h
cert-manager             Active   4h
default                  Active   27h
gpu-feature-discovery    Active   7h31m
k8tz                     Active   7h12m
kube-node-lease          Active   27h
kube-public              Active   27h
kube-system              Active   27h
longhorn-system          Active   10h
metallb-system           Active   22h
monitoring               Active   10h
nfs-provisioning         Active   5h40m
node-feature-discovery   Active   7h32m
nvidia-device-plugin     Active   7h19m
tigera-operator          Active   27h

How to make it work?

Both cert-manager and kube-prometheus-stack in the last version. Kubeadm cluster 1.28.1

What's your helm version?

3.10.1

What's your kubectl version?

1.28.1

Which chart?

kube-prometheus-stack

What's the chart version?

49.2.0

What happened?

Added namespaces to monitor in the helm values, added a servicemonitor in cert-manager namespace, the servicemonitor does not appear in prometheus. The other namespaces work properly.

What you expected to happen?

I would expect the servicemonitor to appear in prometheus

How to reproduce it?

Install the prometheus-stack chart and the cert-manager chart with servicemonitor enabled.

Enter the changed values of values.yaml?

See description

Enter the command that you execute and failing/misfunctioning.

Helm install for both prometheus-stack and cert-manager

Anything else we need to know?

No response

urbaman commented 1 year ago

I do have other namespaces that are not seen by the operator, while for example I can see servicemonitors being scraped in the calico-system namespace:

level=error ts=2023-08-31T10:27:38.57666423Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:27:38.576842289Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:27:38.577992359Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:28:51.419469632Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:28:51.419753567Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:28:51.423015897Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:28:51.485274011Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:28:51.488148063Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:28:51.489000227Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:29:30.49588109Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:29:30.496059599Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T10:29:30.496216362Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"traefik\" does not exist"
level=error ts=2023-08-31T12:27:08.862225817Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:27:08.862519835Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:27:08.863051464Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:30:05.203309512Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:30:05.203297606Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:30:05.206216945Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:30:05.257580897Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:30:05.257588327Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:30:05.25777527Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:31:38.643469956Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T12:31:38.643469683Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-cluster\" does not exist"
level=error ts=2023-08-31T13:40:14.605891254Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:14.606196251Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:14.607577562Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:37.289482013Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:37.289886153Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:37.289959845Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.080113861Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.080180241Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.081910795Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.107670227Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.11002894Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.114511411Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.139029826Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.139214473Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.144408363Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.180854814Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.187309891Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T13:40:39.190238063Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"teleport-agent\" does not exist"
level=error ts=2023-08-31T15:29:12.408959502Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:12.409076932Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:12.409317616Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:12.409389898Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:12.409434538Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:12.409542521Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:12.409593335Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:12.409635664Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:12.409672911Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:30.512461849Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:30.512442587Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:30.513155721Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:30.99188452Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:30.992220067Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:29:30.992721599Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:32:14.481430876Z caller=operator.go:428 component=alertmanageroperator msg="get namespace to enqueue Alertmanager instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:32:14.481654324Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:32:14.482363366Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:32:14.738534751Z caller=operator.go:1120 component=prometheusagentoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
level=error ts=2023-08-31T15:32:14.738548351Z caller=operator.go:991 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"kubernetes-dashboard\" does not exist"
urbaman commented 1 year ago

It was a cache problem, changing the operator deployment to properly check the actual namespaces and restarting it solved.