Open dopey opened 4 years ago
I am experiencing a similar problem, also looking at a /metrics
endpoint.
I was experiencing the very same issue, but when i added these args to the sidecar deployments, the error msg changed from "Unknown Resource" to "target not found"
level=debug ts=2020-05-29T09:38:55.630Z caller=series_cache.go:354 component="Prometheus reader" msg="target not found" labels="{__name__=\"node_namespace_pod:kube_pod_info:\",namespace=\"airflow\",node=\"gke-kubernetes-newsi-kubernetes-443494f1-ccdw\",pod=\"bq-eviction-policy-1589126400-nwhw9\"}"
Any idea why?
In case anyone else is still struggling with this, I see that the example kube-state-metrics deployment scrapes it via service discovery so that it's collected in a 'kubernetes-services-endpoints' Prometheus job.
I added the annotations to my kube-state-metric service and the majority of metrics are now showing up. Some kube_<resource>_labels
metrics are still getting dropped due to too many labels but that seems fixable.
I have
kube-state-metrics
running in thekube-system
namespace and prometheus reading from the/metrics
endpoint using the following configuration in the prometheus-config.yaml:But it seems I may not be doing the relabeling correctly for stackdriver because I'm seeing this error in the sidecar logs:
Any guidance on how I might configure this integration correctly would be much appreciated.