missing kubernetes_io_hostname in metrics

[skuda]

Hello,

In my Kubernetes 1.7.5 cluster, versions:

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T08:56:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T08:56:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

I can't find the label kubernetes_io_hostname for cadvisor metrics, for example:

$ curl http://10.240.255.5:10255/metrics/cadvisor | grep container_network_receive_bytes_total
# HELP container_network_receive_bytes_total Cumulative count of bytes received
# TYPE container_network_receive_bytes_total counter
container_network_receive_bytes_total{id="/docker/e723040e5db85c1caa035a3b5528d0413551b441e0748f762d9e96962434c602",image="gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.7.5",interface="cbr0",name="cranky_mestorf"} 3.248871e+06
container_network_receive_bytes_total{id="/docker/e723040e5db85c1caa035a3b5528d0413551b441e0748f762d9e96962434c602",image="gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.7.5",interface="eth0",name="cranky_mestorf"} 6.9513265082e+10

I am using Prometheus-operator and trying to use this Grafana dashboard: https://grafana.com/dashboards/315

but I have problems because cadvisor doesn't seem to be exporting that information. Is that maybe because of late changes in cadvisor or could be some problem with my configuration?

I opened an issue directly in Prometheus operator here: https://github.com/coreos/prometheus-operator/issues/725, but @brancz showed me there that the issue was related with cadvisor.

Thanks!

[dashpole]

We had to limit the labels that cadvisor adds to metrics because more recent versions of the prometheus client require all time series to have the same set of labels: https://github.com/google/cadvisor/issues/1704 I'm afraid I have no idea where the label originated from; I cant find a reference to it anywhere in the code base. Do you know what the kubernetes_io_hostname label was for?

[skuda]

The value of the label was the hostname of the node where the metric originated from, it was used to filter nodes in Grafana dashboards like the one I linked: https://grafana.com/dashboards/315

The alternative for that is to use the instance label after splitting the port, but it's nicer to see the hostname than the internal cluster network IP.

[Blasterdick]

+1 @skuda that same issue, and for an old ones GKE clusters that label was a must-have one

[shaileshpadave49]

@skuda Can you please provide some sample example how you used the alternative? Instead of kubernetes_io_hostname what can we used to see internal cluster network IP?

[skuda]

Hello @shaileshpadave49,

I don't have access to that cluster anymore but I think the variable is called instance, like this: instance="10.240.255.5:10255", so you can split the string and get the internal IP

[m1o1]

Also curious about this. I'm assuming the kubernetes_io_hostname label was added by cAdvisor by parsing the associated annotation of the node. Not seeing it there (in the prometheus metrics) anymore, so we are going to try using a different field in the Grafana dashboard to filter by Node (I guess the "node" field should work the same way)

[ni-ajardan]

Any updates on this one ? There are numerous dashboards using this label, and they are all hard to use without it. Any plans to get it back, or should we start adjusting the dashboards ? Any links pointing to the deprecation of this label ?

[wajika]

I am surprised that the discussion that existed in 2017 still has no answer to this day. I also want to get an answer. If kubernetes_io_hostname is not used, what should be filled in?

[wajika]

@skuda Did you finally find the answer?

[skuda]

Hey @wajika,

No, at the time, I just started using the internal cluster IP, and later on, I have been using managed kubernetes in cloud providers, using their proposed metrics system, ex: Azure Monitor for AKS.

I think this was a tag that was removed in cadvisor that was being used in some Grafana dashboards, not sure cadvisor replaced it later with something else, I didn't follow up on this issue, sorry.

[leiwingqueen]

I use kube_node_info metric instead which comes from kube-state-metrics. And i need to change the grafana query using the label 'node' nor 'kubernetes_io_hostname'