metrics: correlate Concourse worker metrics to k8s worker node

[cirocosta]

Hey,

Right now it's fairly easy to correlate the metrics that we get from cadvisor with a given node. This makes the process of jumping from one dashboard to the other fairly easy.

That's not true for metrics scraped by kubernetes-pods though - any node-specific information gets dropped during relabelling. As a result, we can't easily filter information by node.

Some possible actions:

• update the relabelling scheme for kubernetes-service-endpoints and go with it, including pod information there (already has node info, misses pod though); or
• update relabelling scheme for kubernetes-pods, including node information there (it already has pod info).

Thanks!

[cirocosta]

Regarding labeling - fixed it here: b0377e98621dcad335405d3db4d42a1317dae143.

This way we can go more align to what the prometheus-operator does, which is, using endpoints from a Service to determine targets.

Given that "by default", kubernetes_node gets dropped, having the relabelling rules controlled by us end up allowing us to have it.