jalateras
commented
5 years ago Here are some more logs
level=debug ts=2018-11-11T07:37:31.707411526Z caller=pod.go:71 component="discovery manager scrape" discovery=k8s role=pod msg="pod update" tg="&targetgroup.Group{Targets:[]model.LabelSet{model.LabelSet{\"__address__\":\"10.16.1.15\", \"__meta_kubernetes_pod_container_name\":\"autoscaler\"}}, Labels:model.LabelSet{\"__meta_kubernetes_pod_host_ip\":\"10.138.0.2\", \"__meta_kubernetes_pod_uid\":\"5d81098e-e2d3-11e8-82a0-42010a8a004d\", \"__meta_kubernetes_pod_annotation_seccomp_security_alpha_kubernetes_io_pod\":\"docker/default\", \"__meta_kubernetes_namespace\":\"kube-system\", \"__meta_kubernetes_pod_ready\":\"true\", \"__meta_kubernetes_pod_node_name\":\"gke-orion-staging-orion-staging-node--c4b2ccb2-7ml9\", \"__meta_kubernetes_pod_label_k8s_app\":\"kube-dns-autoscaler\", \"__meta_kubernetes_pod_label_pod_template_hash\":\"2375374396\", \"__meta_kubernetes_pod_annotation_scheduler_alpha_kubernetes_io_critical_pod\":\"\", \"__meta_kubernetes_pod_name\":\"kube-dns-autoscaler-67c97c87fb-b2wbb\", \"__meta_kubernetes_pod_ip\":\"10.16.1.15\"}, Source:\"pod/kube-system/kube-dns-autoscaler-67c97c87fb-b2wbb\"}"
level=debug ts=2018-11-11T07:37:36.561542994Z caller=client.go:103 component=remote msg="is auth enabled" auth=true url=https://monitoring.googleapis.com:443/
level=debug ts=2018-11-11T07:37:36.613046672Z caller=client.go:152 component=remote msg="sending request to Stackdriver"
level=debug ts=2018-11-11T07:37:36.72144948Z caller=client.go:171 component=remote msg="Partial failure calling CreateTimeSeries" err="rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist)." req="name:\"projects/orion\" time_series:<metric:<type:\"external.googleapis.com/prometheus/go_memstats_stack_sys_bytes\" > resource:<type:\"k8s_node\" labels:<key:\"cluster_name\" value:\"orion-staging\" > labels:<key:\"location\" value:\"us-west1-a\" > labels:<key:\"node_name\" value:\"gke-orion-staging-orion-staging-node--71e82f11-8fgf\" > labels:<key:\"project_id\" value:\"orion\" > > metric_kind:GAUGE value_type:DOUBLE points:<interval:<end_time:<seconds:1541921856 nanos:459000000 > > value:<double_value:3.047424e+06 > > > time_series:<metric:<type:\"external.googleapis.com/prometheus/go_memstats_heap_idle_bytes\" > resource:<type:\"k8s_node\" labels:<key:\"cluster_name\" value:\"orion-staging\" > labels:<key:\"location\" value:\"us-west1-a\" > labels:<key:\"node_name\" value:\"gke-orion-staging-orion-staging-node--71e82f11-8fgf\" > labels:<key:\"project_id\" value:\"orion\" > > metric_kind:GAUGE value_type:DOUBLE points:<interval:<end_time:<seconds:1541921856 nanos:459000000 > > value:<double_value:2.6165248e+07 > > > time_series:<metric:<type:\"external.googleapis.com/prometheus/go_memstats_heap_released_bytes\" > resource:<type:\"k8s_node\" lab
What did you do? Installed the prometheus server on GKE using the instructions here. I then installed an application and annotated the deployment manifest with
prometheus.io/scrape:'true'What did you expect to see? I expected that the prometheus sever to scrape metrics from this pod.
What did you see instead? Under which circumstances? The following logs from the prometheus pod
Environment
I am running this on GKE
1.11.2-gke.15 2