Open utk231 opened 1 year ago
/assign @dgrisonnet /triage accepted
I0725 11:44:17.036335 1 api.go:90] GET http://prometheus.linkerd-viz.svc:9090/api/v1/series?match%5B%5D=request_total%7Bnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1690285397.035 403 Forbidden E0725
This means that prometheus-adapter can't reach your Prometheus instance because it is unauthorized. You most likely need to configure authentication on prometheus-adapter. There are a couple of options that you can use to do that: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/cmd/adapter/adapter.go#L136-L147
+1, I'm also getting the same error when trying to connect to Amazon Managed Service for Prometheus for querying metrics. I am currently using a sigv4-proxy sidecar container for signing the request and a serviceAccount for assuming a role for querying metrics from the AMP Workspace.
@dgrisonnet could you explain what type of auth do we need to configure? Do we need to provide auth details to connect to AWS AMP Workspace or something else. In my limited knowledge, I thought that having an IAM Role that allows querying operations on AWS AMP through a policy is enough for this setup (using serviceAccount annotations).
unable to update list of all metrics: unable to fetch metrics for query "{__name__=\"container:memory_usage_bytes:sum\",container!=\"POD\",kubernetes_namespace!=\"\",pod!=\"\",cluster_name=\"compute\"}": bad_response: unknown response code 403
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
/triage accepted
(org members only)/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
What happened?:
Prometheus adapter not able to get metrics from Linkerd Prometheus
What did you expect to happen?:
I expected the prometheus-adapter to fetch the metrics from custom Linkerd Prometheus in the Kubernetes cluster.
Please provide the prometheus-adapter config:
prometheus: url: http://prometheus.linkerd-viz.svc
port: 9090 path: ""
rules: default: false custom:
Prometheus-adapter logs
I0725 11:44:17.036335 1 api.go:90] GET http://prometheus.linkerd-viz.svc:9090/api/v1/series?match%5B%5D=request_total%7Bnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1690285397.035 403 Forbidden E0725 11:44:17.036384 1 provider.go:229] unable to update list of all metrics: unable to fetch metrics for query "request_total{namespace!=\"\",pod!=\"\"}": bad_response: unknown response code 403
Anything else we need to know?:
Verified the correctness of the prometheus-adapter configuration. Inspected the prometheus-adapter logs with -v=6 around the time the issue occurred. Validated all the manifests and resources related to prometheus-adapter deployment. Verified the version of prometheus-adapter and prometheus. Prometheus adapter deployed without any changes to Service account, Cluster role or Cluster role binding
Environment: https://artifacthub.io/packages/helm/prometheus-community/prometheus-adapter prometheus-adapter version: 0.10.0 Cloud provider or hardware configuration: AWS EKS 1.25