search

openebs / mayastor

Dynamically provision Stateful Persistent Replicated Cluster-wide Fabric Volumes & Filesystems for Kubernetes that is provisioned from an optimized NVME SPDK backend data storage stack.
Apache License 2.0
722 stars 105 forks source link

Node and Volume level capacity metrics #1132

Open giridharshankar013 opened 2 years ago

giridharshankar013 commented 2 years ago

Is your feature request related to a problem? Please describe. Monitoring resource usage is critical for admin users to manage their container infrastructure. In the absence of such metrics, it is difficult to understand the usage pattern and often, the situation turns reactive when the node or volume run out of capacity. It is important to build a mechanism that can help users visualise the capacity consumption and pattern for informed decision making

Describe the solution you'd like Capture the total capacity, used, and allocatable storage per node and per volume at regular intervals and log them. This data is to be processed and made available through the Kubelet summary API and/or prometheus end point for external monitoring systems to consume and take informed decisions

Describe alternatives you've considered None

Additional context None

hiattp commented 1 year ago

Very interested to have these metrics to send into Prometheus/Grafana. I'm new to Mayastor so forgive the ignorance, does v1.0.4 of Mayastor emit metrics of any kind? I've deployed the service monitor described here and looked for the disk pool metrics described here but they don't seem to surface in Prometheus. This addition to the docs implies a slightly different/additional requirement, i.e. setting serviceMonitorSelectorNilUsesHelmValues to false. I've also looked for "standard" PVC monitoring metrics like kubelet_volume_stats_available_bytes and kubelet_volume_stats_capacity_bytes, but those don't seem to emit from Mayastor-backed PVCs either.

Not sure where best to submit this query but given this ticket speaks to planned metrics I'll start here...is there a way in v1 to monitor capacity/usage of Mayastor pools/disks/PVCs or no?

Current Versions: Mayastor tag: mayadata/mayastor:v1.0.4 Kubernetes: Client Version: v1.25.2 Kustomize Version: v4.5.7 Server Version: v1.25.2

tiagolobocastro commented 1 year ago

AFAIK v1 does not have any metrics whatsoever. The current develop does have some very basic disk pool metrics: https://github.com/openebs/mayastor-extensions/blob/develop/exporter/README.md which is what that docs PR is referring to. It should be possible to get the develop pool exporter to work with v1, but I've never tried it.