concourse_workers_containers custom metric issue

[jmmesquitacardoso]

Hi! We're using the customMetric concourse_workers_containers as defined in the chart's example, and we're using prometheus adapter along with it. The issue is that that concourse is calling the custom metric with the app label as concourse-worker, while the metric in Prometheus is being saved while associated with the concourse-web app. For example:

concourse_workers_containers{container="concourse-web",endpoint="prometheus",instance="<instance>",job="concourse-web-prometheus",namespace="<namespace>",platform="linux",pod="concourse-web-5d47f5c978-hmmqf",service="concourse-web-prometheus",worker="concourse-worker-6465dbfb5c-87qww"}

The query being done to the prometheus-adapter is something along these lines:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/:namespace/pods/*/concourse_workers_containers_avg?labelSelector=app=concourse-worker,release=concourse"

This yields no result as the metric is associated with the concourse-web app. Therefore, changing the query to something along these lines retrieves the correct result:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/:namespace/pods/*/concourse_workers_containers_avg?labelSelector=app=concourse-web,release=concourse"

Is there any way in the chart to associate the concourse_workers_containers metric with the concourse-worker app so it can be successfully fetched?

Thanks!

[jfrancisco0]

I think that label, app=concourse-worker, may be coming from the HPA

[joaosa]

I am also very interested in knowing how to work around this :)

[joaosa]

So it turns out there's a way to work around this with metric relabelling:

            metricRelabelings:
            - action: replace
              sourceLabels:
              - worker
              targetLabel: pod

This way, the pod label gets replaced with the worker one, which points to concourse-worker pods. Consequently, the concourse worker HPA is now able to fetch custom metrics, as they are now associated to the concourse-worker pods and not the concourse-web one anymore.