kais271 / pvc-exporter

Provides metrics to monitoring mounted pvc usage.
Apache License 2.0
43 stars 18 forks source link

No block storage pvc found or not supported yet #20

Closed saingithub closed 2 years ago

saingithub commented 3 years ago

In my kubernetes environment Dell CSI driver is the storage provisioner. When I implemented this PV-exporter I am getting the following error in the logs No block storage pvc found or not supported yet

kais271 commented 3 years ago

In my kubernetes environment Dell CSI driver is the storage provisioner. When I implemented this PV-exporter I am getting the following error in the logs No block storage pvc found or not supported yet

@saingithub Could you please share the result for "df -h" ? and tell me which version you using.

saingithub commented 3 years ago

Thanks, @kais271, It worked for me. Could you please let me know how can I add it in Grafana Dashboard

kais271 commented 3 years ago

@saingithub which version is you using ? If you using v0.1.2, you can try use this query "(sum without (container,pod,service,namespace,job,instance,endpoint) (pvc_usage)) + on(volumename) group_left(persistentvolumeclaim,mountedby,pod_namespace)pvc_mapping*0 "

saingithub commented 3 years ago

The version i am using is v7.1.5. When I give that query I am getting the following error.

"invalid expression type \"string\" for range query, must be Scalar or instant Vector"

kais271 commented 3 years ago

The version i am using is v7.1.5. When I give that query I am getting the following error.

"invalid expression type "string" for range query, must be Scalar or instant Vector"

don't have "" ..just *(sum without (container,pod,service,namespace,job,instance,endpoint) (pvc_usage)) + on(volumename) group_left(persistentvolumeclaim,mountedby,pod_namespace)pvc_mapping0**

saingithub commented 3 years ago

"many-to-many matching not allowed: matching labels must be unique on one side"

kais271 commented 3 years ago

could you please share the full error? and check whether they have the same "job"

saingithub commented 3 years ago

Is there is any option to get it in a tabular format in Grafana

kais271 commented 3 years ago

Sorry, I don't know

saingithub commented 3 years ago

I tried deploying v0.1.3-alpha version. I am getting the following error in the pod log

File "./pvc-exporter.py", line 9, in EXPORTER_SERVER_PORT=int(os.getenv('EXPORTER_SERVER_PORT')) TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

kais271 commented 3 years ago

you can check if the yaml of the pod has an EXPORTER_SERVER_PORT variable

saingithub commented 3 years ago

I had used the yaml file of v0.1.2 for the deployment of the pod.

kais271 commented 3 years ago

that's not work. you need use v0.1.3-alpha chart to install v0.1.3-alpha. because it has more changes.

saingithub commented 2 years ago

In v0.1.3 you have now created deployment files in which lots of variables are used compared to the previous version. How can I interpret that to deploy in a Kubernetes environment

kais271 commented 2 years ago

@saingithub There are 3 variables: HOST_IP,EXPORTER_SERVER_PORT,SCAN_INTERVAL. HOST_IP: This used to match pv. We get all pv via API, and deploy pvc-exporter as daemonset. So the pod only get the pv size of hostpath pv on current the node. EXPORTER_SERVER_PORT: This is exporter http server port in pod. SCAN_INTERVAL: This is scan pvc interval, unit is second.

saingithub commented 2 years ago

grafana

I have more than 100 PVC. I have attached the grafana dashboard image. Is there any way to filter it more so that we can interpret it easy.

kais271 commented 2 years ago

@saingithub You can try this dashsboard. It can filter by namespace or pod. https://github.com/kais271/pvc-exporter/blob/v0.1.3-alpha/docs/PVC%20%20Dashboard-1637745492005.json

image

saingithub commented 2 years ago

When I used the above file I am getting the following error in Grafana

Panel plugin not found: timeseries

kais271 commented 2 years ago

Maybe caused by grafana version. My version is v8.1.2 . Try the following steps

  1. (sum without (container,pod,service,namespace,job,instance,endpoint,pvc_namespace,pvc_requested_size_MB) (pvc_usage)) + on(persistentvolume) group_left(persistentvolumeclaim,mountedby,pod_namespace)pvc_mapping{pod_namespace=~"$pod_namespace",mountedby=~"$Pod"}*0 image 2. image 3. image

saingithub commented 2 years ago

When i give the above expression no data is getting in the dashboard grafana1

saingithub commented 2 years ago

Hi, Thanks, @kais271.

It worked for me. Now I was able to differentiate PVC usage by namespace and pods.

One more thing , is there any way to create an alert mechanism if any PVC exceeds a specified limit

kais271 commented 2 years ago

@saingithub You can reference this: https://grafana.com/docs/grafana/latest/alerting/old-alerting/create-alerts/ But "Template variables are not supported in alert queries", so you need to create a new dashboard for alert.

kais271 commented 2 years ago

Issues close after 30d of inactivity. Reopen the issue with /reopen.