search

prometheus-community / stackdriver_exporter

Google Stackdriver Prometheus exporter
Apache License 2.0
261 stars 98 forks source link

Timeout fetching Google stackdriver monitoring metrics (Client.Timeout exceeded while awaiting headers) #152

Open pcperera opened 2 years ago

pcperera commented 2 years ago

Issue

Stackdriver exporter throws the below error when fetching stack driver monitoring metrics.

{ labels: { compute.googleapis.com/resource_name: "gke-company-resource-name" k8s-pod/app: "prometheus-stackdriver-exporter" k8s-pod/release: "stable" } logName: "projects/company/logs/stderr" receiveTimestamp: "2022-04-20T01:15:51.291425040Z" resource: { labels: { cluster_name: "company-cluster1" container_name: "prometheus-stackdriver-exporter" namespace_name: "internal-services" pod_name: "stable-prometheus-stackdriver-exporter-pod-suffix" project_id: "company-project-id" } type: "k8s_container" } severity: "ERROR" textPayload: "level=error ts=2022-04-20T01:15:49.913Z caller=monitoring_collector.go:185 msg="Error while getting Google Stackdriver Monitoring metrics" err="Get \"https://monitoring.googleapis.com/v3/projects/company-cluster1/metricDescriptors?alt=json&filter=metric.type+%3D+starts_with%28%22compute.googleapis.com%2Finstance%2Fcpu%22%29&prettyPrint=false\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"" timestamp: "2022-04-20T01:15:49.913872755Z" }

Question

What is the timeout parameter that has to increased to prevent timing out ? Below Helm chart is used to deploy stack driver exporter.

image

xbglowx commented 2 years ago

Dup of https://github.com/prometheus-community/stackdriver_exporter/issues/97 ?

automaticgiant-ml commented 2 years ago

perhaps it's this? https://github.com/prometheus-community/stackdriver_exporter/blob/master/stackdriver_exporter.go#L60-L62