This PR introduces the alpha version of a persistent volume capacity monitor container for Kubernetes. It is typically expected to be deployed as a sidecar to the node-exporter on the cluster. The container uses the textfile collector approach to gather the PV metrics, while allowing node-exporter to expose them. The flow/working & usage details of the collector is provided below:
The pod running this container is expected to use a service account with cluster-level scope. (In case of the openebs node-exporter: openebs-maya-operator)
The pv-monitor sidecar is expected to have similar volume mounts as the node-exporter, including the host root fs configured with a HostToContainer propagation to detect new PV mounts on the host.
A list of active/Bound PVs is obtained, which capacity/size & current utilization are gathered. The utilization is obtained via the du command on the PVs mounted at the /var/lib/kubelet/pods/<> paths
A set of of textfiles are updated into a specified shared path (achieved by a shared temp volume such as emptyDir{}), via a polling mechanism, with configurable intervals in a format desired by the node-exporter:
The node-exporter is configured to monitor the textfile collector directories.
--collector.textfile.directory=/shared_vol
Which issue this PR fixes(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Subsequent improvements can include ability to replace values on the textfiles instead of recreate, using a lighter base image & making the scripts more robust for failure conditions (change to python/go). Other approaches to expose the kubelet_volume_stats metrics on the individual nodes (instead of du/df operations) will also be considered.
Sample dashboard showing utilization percentages with the prom query: ((pv_utilization_bytes) *100 / pv_capacity_bytes)
Signed-off-by: ksatchit karthik.s@openebs.io
What this PR does / why we need it:
This PR introduces the alpha version of a persistent volume capacity monitor container for Kubernetes. It is typically expected to be deployed as a sidecar to the node-exporter on the cluster. The container uses the textfile collector approach to gather the PV metrics, while allowing node-exporter to expose them. The flow/working & usage details of the collector is provided below:
The pod running this container is expected to use a service account with cluster-level scope. (In case of the openebs node-exporter: openebs-maya-operator)
The pv-monitor sidecar is expected to have similar volume mounts as the node-exporter, including the host root fs configured with a HostToContainer propagation to detect new PV mounts on the host.
A list of active/Bound PVs is obtained, which capacity/size & current utilization are gathered. The utilization is obtained via the
ducommand on the PVs mounted at the/var/lib/kubelet/pods/<>pathsA set of of textfiles are updated into a specified shared path (achieved by a shared temp volume such as emptyDir{}), via a polling mechanism, with configurable intervals in a format desired by the node-exporter:
Which issue this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close that issue when PR gets merged): fixes #Special notes for your reviewer:
Subsequent improvements can include ability to replace values on the textfiles instead of recreate, using a lighter base image & making the scripts more robust for failure conditions (change to python/go). Other approaches to expose the kubelet_volume_stats metrics on the individual nodes (instead of du/df operations) will also be considered.
Sample dashboard showing utilization percentages with the prom query:
((pv_utilization_bytes) *100 / pv_capacity_bytes)