search

keegancsmith / kubernetes-disk-exporter

Export kubernetes persistent volume metrics to prometheus.
MIT License
9 stars 3 forks source link

Disks from rescheduled pods are not exported #1

Open stvnwrgs opened 7 years ago

stvnwrgs commented 7 years ago

It looks like, that the disks for a pod that gets scheduled after the disk-exporter are not exported. If I delete the disk-exporter and the disk-exporter pod gets created again by the daemonset, it catches the new disks.

Steps to reproduce:

  1. create disk-exporter pod
  2. create pod
  3. check metrics

I already looked into the code and expected something like an initial scan of the mounts, but it looks like that this is not the case. At least if Prometheus is executing the Collect function every time the handler runs.

Any Ideas?

keegancsmith commented 7 years ago

Hey @stvnwrgs, yeah unfortunately this is an issue I have noticed in production as well. I think its more to do with how docker or the kubelet sets up the disk-exporter pod. My educated guess is it takes a snapshot of mounts at the pod start time and does not update them as the hostpath changes.

I haven't made the time to follow up yet. I'll file an issue on the kubernetes repo. In the meantime I can add the awful hack of making the pod restart itself every 5min.

stvnwrgs commented 7 years ago

Hey @keegancsmith, if this is the case, then this is definitely a bug in docker or kubernetes.

A restart could be a hacky fix for this. It would increase the restart count, but at least you would get metrics for your disks. I think this is more important than the restarts counter. Maybe you could add an optional argument with a restart time until the main reason is found?

keegancsmith commented 7 years ago

@stvnwrgs latest master + the latest docker image include the -short-lived flag to work around this.