sas-go-pods podmonitor incorrect

[tynsh]

Hi,

we've had an issue with the sas-go-pods podmonitor. It tries to scrape all container ports exposed and not just the ones specified in the labels. Here is the relevant part from the prometheus targets page.

https://10.244.11.43:9999/internal/metrics | DOWN | container="sas-report-operations-core"endpoint="http"instance="10.244.11.43:9999"job="sas-report-operations"namespace="viya"node="aks-stateless-43847504-vmss000003"pod="sas-report-operations-7485dc55f6-fmmtb"sas_deployment="sas-viya"sas_service_base="golang"service="sas-report-operations" | 22.792s ago | 2.089ms | Get "https://10.244.11.43:9999/internal/metrics": http: server gave HTTP response to HTTPS client -- | -- | -- | -- | -- | -- http://10.244.11.38:9114/internal/metrics | DOWN | container="sas-opendistro-exporter"endpoint="http"instance="10.244.11.38:9114"job="sas-opendistro-exporter"namespace="viya"node="aks-stateless-43847504-vmss000003"pod="sas-opendistro-exporter-5bcc7b79c5-nmp2f"sas_deployment="sas-viya"sas_service_base="golang"service="sas-opendistro-exporter" | 1.885s ago | 0.952ms | expected a valid start token, got "<" ("INVALID") while parsing: "<" https://10.244.11.85:8080/internal/metrics | DOWN | container="sas-redis-operator"endpoint="http"instance="10.244.11.85:8080"job="sas-redis-operator"namespace="viya"node="aks-stateless-43847504-vmss000003"pod="sas-redis-operator-8487f55959-49zpr"sas_deployment="sas-viya"sas_service_base="golang"service="sas-redis-operator" | 23.228s ago | 1.359ms | Get "https://10.244.11.85:8080/internal/metrics": dial tcp 10.244.11.85:8080: connect: connection refused

We have added the following to the podMonitor spec:

    - action: replace
      regex: true;(.+)
      replacement: $1
      sourceLabels:
      - __meta_kubernetes_pod_annotationpresent_prometheus_io_port
      - __meta_kubernetes_pod_annotation_prometheus_io_port
      targetLabel: __port
    - action: replace
      regex: (.+);(.+)
      replacement: $1:$2
      sourceLabels:
      - __meta_kubernetes_pod_ip
      - __port
      targetLabel: __address__
    - action: replace
      regex: true;(.+)
      replacement: $1
      sourceLabels:
      - __meta_kubernetes_pod_annotationpresent_prometheus_io_path
      - __meta_kubernetes_pod_annotation_prometheus_io_path
      targetLabel: __metrics_path__

In the same podMonitor spec we also added sas-redis-operator to the excluded list of pods. Even though the Kubernetes Yaml specifies a port for scraping (8080), inside the pod no TCP Port is opened.

    - sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
      regex: "sas-redis-operator|sas-arke|sas-launcher"
      action: drop

Hope you find this helpful.

Best

Tobias Wackenhut

[gsmith-sas]

@tynsh Thank you for submitting this. Our team will review your information and respond here.

[tynsh]

Just a quick note. This is on the last Viya LTS release. (First one in 2023, so 2023.03?)