Open pmalek opened 6 months ago
Hello @pmalek , https://docs.datadoghq.com/containers/kubernetes/prometheus/?tab=kubernetesadv2 is the more up-to-date guide to follow for Prometheus/Openmetrics collection in Kubernetes environments.
The bearer_token_auth
is indeed exclusive to the prometheus_url
implementation / class of the Openmetrics check. Nonetheless, you can replicate the same behaviour with the auth_token
configuration parameter : https://github.com/DataDog/integrations-core/blob/master/openmetrics/datadog_checks/openmetrics/data/conf.yaml.example#L424-L461. More specifically, your configuration with openmetrics_endpoint
would be :
auth_token:
reader:
type: file
path: /var/run/secrets/kubernetes.io/serviceaccount/token
writer:
type: header
name: Authorization
value: Bearer <TOKEN>
This would allow you to use openmetrics_endpoint
implementation which is more performant than the legacy prometheus_url
implementation which was created from https://docs.datadoghq.com/integrations/prometheus/ while using the Kubernetes service account scope.
Thanks for the reponse @tbavelier.
Sadly https://docs.datadoghq.com/containers/kubernetes/prometheus/?tab=kubernetesadv2 doesn't mention anything about the auth tokens 😅 But it's good to know that we can use that. I'll check this next time I'm around DD configuration.
Additional environment details (Operating System, Cloud provider, etc):
Kubernetes, kind
Problem statement
https://docs.datadoghq.com/integrations/guide/prometheus-host-collection/#parameters-available mentions the usage of
bearer_token_path
,bearer_token_auth
andopenmetrics_endpoint
but noprometheus_url
, in order to make the agent attach ServiceAccount's token to requests.After some testing and several hours of debugging I came to https://github.com/DataDog/integrations-core/blob/14f280a0f339e53a7703c86d687a8ca32aa683ed/datadog_checks_base/datadog_checks/base/checks/openmetrics/base_check.py#L15 and several other lines in this file which got be thinking that I should use
prometheus_url
.That's exactly what was missing. I switched
openmetrics_endpoint
toprometheus_url
and now my Pods are correctly attaching the token to requests.Possibly related
https://github.com/DataDog/datadog-agent/issues/2667