search

openebs / monitoring

OpenEBS Monitoring add-on. A set of Grafana, Prometheus, and alert manager plugins.
Apache License 2.0
31 stars 20 forks source link

Bug: NDM dashboard doesn't show metrics when ndm is installed via helm #79

Closed kmova closed 2 years ago

kmova commented 3 years ago

Ref: https://github.com/openebs/monitoring/blob/develop/jsonnet/config.libsonnet#L265

NDM Service Monitor is setup with matching labels name: openebs-ndm-exporter

Depending on how the NDM is installed ( kubectl vs helm), the label name is different.

This results in NDM metrics not being reported when installed through the helm.

To fix the above - use a common label from both installations. Change the NDM service monitor to use component. 

component: 'openebs-ndm-node-exporter'
Ab-hishek commented 3 years ago

This would only help get metrics from ndm node exporter daemonset and not cluster-exporter single deployment. Better to fix this issue from ndm itself. When installed via ndm-operator(ore kubectl), we are using a common label for both the exporters, this makes a single service monitor from the monitoring front to collect metrics from both the exporters. But if we change the service in the service monitor itself then we will have to create two separate monitors for node and cluster exporter. Therefore, better to include the common label during the ndm helm installation also as it used by the operator. The common label used for both the exporters are: name: openebs-ndm-exporter

Ab-hishek commented 3 years ago

This label can be included in the labels section used by ndm node-exporter and cluster-exporter services helm templates-> https://github.com/openebs/node-disk-manager/blob/a1b308a9801dd4ed816e193352ded4b7fd341cb9/deploy/helm/charts/templates/_helpers.tpl#L240 and https://github.com/openebs/node-disk-manager/blob/a1b308a9801dd4ed816e193352ded4b7fd341cb9/deploy/helm/charts/templates/_helpers.tpl#L213

Ab-hishek commented 3 years ago

@kmova I see that is already taken care of by the https://github.com/openebs/node-disk-manager/blob/a1b308a9801dd4ed816e193352ded4b7fd341cb9/deploy/helm/charts/templates/_helpers.tpl#L232 if the helm installation takes place during the default values for the exporter. If the values is changed then we need to change the service monitors labels as well. Can you show the ndm service and ndm node and cluster exporter labels for helm installation?

Ab-hishek commented 3 years ago

Upon investigation, found that there is values configured in values.yaml for ndm-node-exporter and ndm-cluster-exporter(https://github.com/openebs/node-disk-manager/blob/a1b308a9801dd4ed816e193352ded4b7fd341cb9/deploy/helm/charts/values.yaml#L94 & https://github.com/openebs/node-disk-manager/blob/a1b308a9801dd4ed816e193352ded4b7fd341cb9/deploy/helm/charts/values.yaml#L102) respectively which overrides the common exporter label name: openebs-ndm-exporter while the templates are being formed. We need to update the podlabels values in the values.yaml with the common label. Everything should work fine then.