search

concourse / hush-house

Concourse k8s-based environment
https://hush-house.pivotal.io
29 stars 23 forks source link

metrics: add panel for worker container file descriptors #18

Open cirocosta opened 5 years ago

cirocosta commented 5 years ago

Hey,

Despite the various panels that we have, one that we miss for the worker containers is "Number of open files" and "Number of processes".

Right now, we have a similar one for the web pods given that the golang Prometheus client library exposes that through process_*, but given that the worker does not have an exporter, we'd need to rely on cAdvisor for that.

Unfortunately, with the current version of k8s in GKE (v1.12.5), the latest cAdvisor is not used - what we have there is 0.30.1 see release notes, while the metric we need is only in 0.32.0.

cirocosta commented 5 years ago

The 1.13 series of Kubernetes will include this metric! https://github.com/kubernetes/kubernetes/pull/70964

cirocosta commented 5 years ago

As we're using 1.13 there right now, it might be a good time to come back to this issue 😁

(even though I'd much rather have worker exposing prometheus metrics by itself 😬 )