dashpole
commented
4 years ago What do you mean by "unnamed" resources? You should check your prometheus relabeling config. Prometheus may be intentionally dropping some metrics.
Open neilharris123 opened 4 years ago
dashpole
commented
4 years ago What do you mean by "unnamed" resources? You should check your prometheus relabeling config. Prometheus may be intentionally dropping some metrics.
neilharris123
commented
4 years ago What do you mean by "unnamed" resources? You should check your prometheus relabeling config. Prometheus may be intentionally dropping some metrics.
By unnamed resources, I mean there is no name displayed for a specific pod which is generating metrics (the one in the image whose the information is displayed).
The rest of the pods represented by lines on the graph do have have their pod name displayed. For example:
The metrics are not scraped by prometheus, they are scraped by datadog, and there is no re-labelling in the configuration.
dashpole
commented
4 years ago I believe metrics with pod set, but container not set are pod-level cgroup metrics.
When I generate a timeseries graph in datadog using a cadvisor metrics (for example 'container_memory_usage_bytes',) from a cadvisor daemonset I have deployed, the resulting chart displays metrics for apparently un-named resources- both pods and namespaces:
When I check the same metric using prmetheus/grafana, there is no un-named resources- however, prometheus uses the cadvisor metrics from the kubelet. This should be impossible as you cannot have un-named resources in kubernetes. Is this a bug with cadvisor daemonset?