I am unable to find the prometheus rules

[Evantage-WS]

Hi,

I have installed the helm with:

helm repo add enix https://charts.enix.io
helm repo update

NAMESPACE="x509-certificate-exporter"

helm upgrade x509-certificate-exporter enix/x509-certificate-exporter -f monitor-host-values.yaml \
    --install \
    --create-namespace \
    --namespace ${NAMESPACE}

The monitor-host-values.yaml contains:

hostPathsExporter:
  daemonSets:
    cp:
      nodeSelector:
        node-role.kubernetes.io/master: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      watchFiles:
      - /var/lib/kubelet/pki/kubelet-client-current.pem
      - /etc/kubernetes/pki/apiserver.crt
      - /etc/kubernetes/pki/apiserver-etcd-client.crt
      - /etc/kubernetes/pki/apiserver-kubelet-client.crt
      - /etc/kubernetes/pki/ca.crt
      - /etc/kubernetes/pki/front-proxy-ca.crt
      - /etc/kubernetes/pki/front-proxy-client.crt
      - /etc/kubernetes/pki/etcd/ca.crt
      - /etc/kubernetes/pki/etcd/healthcheck-client.crt
      - /etc/kubernetes/pki/etcd/peer.crt
      - /etc/kubernetes/pki/etcd/server.crt
      watchKubeconfFiles:
      - /etc/kubernetes/admin.conf
      - /etc/kubernetes/controller-manager.conf
      - /etc/kubernetes/scheduler.conf
    nodes:
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/ingress
        operator: Exists
      watchFiles:
      - /var/lib/kubelet/pki/kubelet-client-current.pem
      - /etc/kubernetes/pki/ca.crt

But I am unable to find the Prometheus rules. there is nothing in Prometheus (rules or alerts) with x509. I am running Kube-Prometheus-Stack 20.0.1 (Helm) with K8s 1.22.2

Can you help me out with this? Thanks!

[npdgm]

Hi @ws-prive

Given the steps you followed for the installation, I assume you do have the PrometheusRules resource created by the chart and that it is not an issue here.

Four possible explanations come to mind:

• Your cluster may have RBAC rules that prevent the Prometheus operator to list and read the PrometheusRule. Investigating operator logs will show evidence of that. I would watch it's logs while for example recreating the PrometheusRule by uninstalling then installing the exporter chart again.
• Perhaps the kube-prometheus-stack was configured with restrictions on what Prometheus operator custom resources are enumerated and from which namespaces. By default if would pick them all. Again, operator logs would hint whether this is the case or not. Could you show a dump of the Prometheus resource? kubectl -n ${PROMETHEUS_STACK_NAMESPACE} get prometheus prometheus-kube-prometheus-prometheus -o yaml In particular the serviceMonitorNamespaceSelector: and serviceMonitorSelector: keys.
• It is possible either Prometheus failed to reload the configuration, or the operator is stuck on invalid content. Which could be from any custom resource of this chart or other apps. Admission webhook are supposed to validate PrometheusRules but I guess there still could be an issue on that side. If you can't find any error or activity in the Prometheus Pod logs or at the operator, perhaps they should both be restarted.
• Finally there might be an issue with the Prometheus operator and Kubernetes 1.22 that I'm not aware of yet. We tend to lag one or two minor versions for compatibility reasons with other apps in clusters where this exporter is developed. It seems unlikely to be a source for trouble, but should you not find a fix I will try to replicate your environment.

Let us know what you can find in the operator logs.

Cheers

[Evantage-WS]

Hi @npdgm,

Thanks for the reply. I can't find a single line in the logs related to the install. The only item I can find in the PrometheusRules CRD's is that it is installed in the namespace I have given (x509-certificate-exporter). I have also tested it with an install in the same namespace as Prometheus, but no luck. I even restarted all the pods related to Prometheus, but this didn't help either.

I am lost with your question about the dump, you stated kubectl -n monitoring get prometheus prometheus-kube-prometheus-prometheus -o yaml, but this doesn't work. I have looked if I can find what you need, but it remains unclear. Could you please tell me how to get your dump? I am using the kube-prometheus-stack.

Thanks!

[npdgm]

Alright so we really should have a look at the Prometheus custom resource that was created by kube-prometheus-stack. The command I gave may not work if the release name was customized. To find it you may try: kubectl -n monitoring get prometheus or kubectl get -A prometheus There should be a single one unless your kube distro or tools such as Rancher also deployed a monitoring stack. Then you get the resource in YAML output and we can have a look at serviceMonitorSelector: and other settings that could explain your issue.

Cheers

[Evantage-WS]

Hi @npdgm,

Thanks, this is the output of kubectl -n monitoring get prometheus kube-prometheus-stack-prometheus -o yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  annotations:
    meta.helm.sh/release-name: kube-prometheus-stack
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2021-11-18T06:55:22Z"
  generation: 1
  labels:
    app: kube-prometheus-stack-prometheus
    app.kubernetes.io/instance: kube-prometheus-stack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 20.0.1
    chart: kube-prometheus-stack-20.0.1
    heritage: Helm
    release: kube-prometheus-stack
  name: kube-prometheus-stack-prometheus
  namespace: monitoring
  resourceVersion: "2168"
  uid: e885d9c9-4d8b-45c6-b838-d5ec709745f6
spec:
  alerting:
    alertmanagers:
    - apiVersion: v2
      name: kube-prometheus-stack-alertmanager
      namespace: monitoring
      pathPrefix: /
      port: web
  disableCompaction: true
  enableAdminAPI: false
  externalLabels:
    clustername: k8s-central
    geo: nl
    region: nb
  externalUrl: http://kube-prometheus-stack-prometheus.monitoring:9090
  image: quay.io/prometheus/prometheus:v2.28.1
  listenLocal: false
  logFormat: logfmt
  logLevel: info
  paused: false
  podMonitorNamespaceSelector: {}
  podMonitorSelector:
    matchLabels:
      release: kube-prometheus-stack
  portName: web
  probeNamespaceSelector: {}
  probeSelector:
    matchLabels:
      release: kube-prometheus-stack
  replicas: 1
  retention: 6h
  routePrefix: /
  ruleNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      app: kube-prometheus-stack
      release: kube-prometheus-stack
  secrets:
  - etcd-client-cert
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: kube-prometheus-stack-prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  shards: 1
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 2Gi
  thanos:
    objectStorageConfig:
      key: objstore.yml
      name: thanos-objstore-config
  version: v2.28.1

I installed the prometheus-stack with this option, could this be the issue?

 --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \

[Evantage-WS]

Hi @npdgm,

Tested without the --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false, but result is the same.

[npdgm]

serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}

This is fine and shall let the Prometheus operator find ServiceMonitors in all namespaces.

Do you get "yes" when running this?

kubectl auth can-i get servicemonitor -n x509-certificate-exporter --as system:serviceaccount:monitoring:kube-prometheus-stack-operator

[Evantage-WS]

Hi @npdgm,

Did you remove the line in your reply: Don't set value ...serviceMonitorSelectorNilUsesHelmValues=false as this would basically disable the use of ServiceMonitor resources, and/or restrict lookups to the operator's own namespace. ? I have tested it with both, so I removed that line from the helm install and I did an install with that line. Both times the output is yes.

Output of

Release "x509-certificate-exporter" does not exist. Installing it now.
NAME: x509-certificate-exporter
LAST DEPLOYED: Thu Nov 18 13:35:57 2021
NAMESPACE: x509-certificate-exporter
STATUS: deployed
REVISION: 1
TEST SUITE: None
❯ kubectl auth can-i get servicemonitor -n x509-certificate-exporter --as system:serviceaccount:monitoring:kube-prometheus-stack-operator
yes
╭─ ~/Dow/D/Kubernetes & Docker/Git/kubernetes-labs/cert-x509-certificate-exporter on main ⇡1 *1 +1 !9 ?4                                                                                               
╰─❯ 

If you want I can give you access to my labmachine if that's helps you.

[npdgm]

Did you remove the line in your reply: Don't set value ...serviceMonitorSelectorNilUsesHelmValues=false as this would basically disable the use of ServiceMonitor resources, and/or restrict lookups to the operator's own namespace. ? I have tested it with both, so I removed that line from the helm install and I did an install with that line. Both times the output is yes.

Indeed, I removed the comment as you where only performing a test and I misread for a second.

Ok I can have a look at your system if you send some credentials to root@@devcat.org

Cheers

[Evantage-WS]

Hi @npdgm,

If I do understand you correctly, the line --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false is not an issue and can remain in the helm install.

I send you a mail regarding the remote access.

[Evantage-WS]

Hi @npdgm,

Did you receive my mail?

[Evantage-WS]

Hi @npdgm,

Any update on this? Thanks!

[Evantage-WS]

I solved it, to help others, the solution is to use this in the prometheus-operator(-stack) helm install:

  --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false \