kube-state-metrics "unkown resource"

[dopey]

I have kube-state-metrics running in the kube-system namespace and prometheus reading from the /metrics endpoint using the following configuration in the prometheus-config.yaml:

      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

But it seems I may not be doing the relabeling correctly for stackdriver because I'm seeing this error in the sidecar logs:

level=debug ts=2020-03-09T22:32:07.883Z caller=series_cache.go:361 component="Prometheus reader" msg="unknown resource" labels="{instance=\"kube-state-metrics.kube-system.svc.cluster.local:8080\", job=\"kube-state-metrics\"}" discovered_labels="{__address__=\"kube-state-metrics.kube-system.svc.cluster.local:8080\", __metrics_path__=\"/metrics\", __scheme__=\"http\", _kubernetes_cluster_name=\"primary\", _kubernetes_location=\"us-central1\", _stackdriver_project_id=\"<redacted>\", job=\"kube-state-metrics\"}"

Any guidance on how I might configure this integration correctly would be much appreciated.

[funzie19]

I am experiencing a similar problem, also looking at a /metrics endpoint.

[geoashis]

I was experiencing the very same issue, but when i added these args to the sidecar deployments, the error msg changed from "Unknown Resource" to "target not found"

• --stackdriver.generic.location=us-east1-b
• --stackdriver.generic.namespace=cluster-name

level=debug ts=2020-05-29T09:38:55.630Z caller=series_cache.go:354 component="Prometheus reader" msg="target not found" labels="{__name__=\"node_namespace_pod:kube_pod_info:\",namespace=\"airflow\",node=\"gke-kubernetes-newsi-kubernetes-443494f1-ccdw\",pod=\"bq-eviction-policy-1589126400-nwhw9\"}"

Any idea why?

[parthmishra]

In case anyone else is still struggling with this, I see that the example kube-state-metrics deployment scrapes it via service discovery so that it's collected in a 'kubernetes-services-endpoints' Prometheus job.

I added the annotations to my kube-state-metric service and the majority of metrics are now showing up. Some kube_<resource>_labels metrics are still getting dropped due to too many labels but that seems fixable.