Empty custom metrics from Prometheus Adapter, AWS EKS

[sweetdandelions]

The goal was to deploy a Groundplex (SnapLogic) in Kubernetes following the documentation https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/2017558845/Deploying+a+Groundplex+in+Kubernetes+with+Elastic+Scaling

Everything works up until the Prometheus Adapter not being able to scrape the custom metrics defined in the yaml file.

prometheus:
  # Run `kubectl get svc -n namespace` to get the url of Prometheus
  url: http://prometheus-kube-prometheus-prometheus.monitoring.svc
  # Prometheus's port. Currently it is LoadBalancer with 80 default.
  port: 80
replicas: 1
rules:
  # Whether to enable the default rules
  default: false
  # Add custom metrics API here
  custom:
  # plex_queue_size API
  - seriesQuery: 'plex_queue_size'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: ^(.*)
      as: "plex_queue_size"
    metricsQuery: <<.Series>>{<<.LabelMatchers>>}

Expected output:

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "pods/plex_queue_size",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "namespaces/plex_queue_size",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

What I got:

{
    "kind":"APIResourceList",
    "apiVersion":"v1",
    "groupVersion":"custom.metrics.k8s.io/v1beta1",
    "resources": []
}

There have been similar topics regarding the issue, but I couldn't find any solution for mine. Not an expert in all these topics, started learning a while ago.

Versions used:

$ kubectl version --short
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-14T19:49:27Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
$ helm version
version.BuildInfo{Version:"v3.12.1", GitCommit:"f32a527a060157990e2aa86bf45010dfb3cc8b8d", GitTreeState:"clean", GoVersion:"go1.20.4"}

Latest Prometheus/Adapter versions. I'm using AWS EKS, infrastructure can be found in the repo https://github.com/sweetdandelions/kube-app-slp

Any help or suggestions on the topic is appreciated.

[dgrisonnet]

/kind support /triage accepted /assign

[VarunTyagi86]

Any update on this issue, we are facing the same issue.

[dudeitssm]

I'm also seeing the same issue.

For reference, I am using the following guide.

I have double checked that:

1. I have connectivity to the prometheus service via both local (k8s) DNS as well as via internet.
2. I also have included http:// in the URL.

[dilinade]

I'm facing the same issue. Response for kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": []
}

[danparisi]

Same issue here. Additionally, I noticed the docs are mentioning different syntax about resource overrides section in values.yam. In some place I saw:

      resources:
        overrides:
          namespace: { resource: "namespace" }
          pod: { resource: "pod" }

I.e.: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/sample-config.yaml Somewhere else:

      resources:
        overrides:
          kubernetes_namespace: { resource: "namespace" }
          kubernetes_pod_name: { resource: "pod" }

I.e.: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config-walkthrough.md

Or maybe the _kubernetes__ prefix should be used only for custom rules, while no prefix is required for other rule types? It's unclear.

[danparisi]

On my side, I fixed the issue. The Prometheus adapter was not able to connect to the prometheus server. You only realize it by looking for such logs:

E0723 13:53:52.921151       1 provider.go:229] unable to update list of all metrics: unable to fetch metrics for query ...

About the syntax to be used in values.yaml, this is the right one:

      resources:
        overrides:
          namespace: { resource: "namespace" }
          pod: { resource: "pod" }