search

dell / csm

Dell Container Storage Modules (CSM)
Apache License 2.0
71 stars 16 forks source link

I have setup the CSM Observability Module and ist working great for powerscale, but powerstore is not reporting anything? #740

Closed DTW2023 closed 1 year ago

DTW2023 commented 1 year ago

How can the Team help you today?

PowerStore 500T running 3.0.0.1

I have CSM Observability module installed and its working great with powerscale, but grafana isn't displaying anything for powerstore. Any suggestiosn on what to check?

donatwork commented 1 year ago

have you gone through all of the post-install tasks? Did you add Prometheus as a data source? Did you add the Grafana dashboards. https://dell.github.io/csm-docs/docs/observability/deployment/#post-installation-dependencies

DTW2023 commented 1 year ago

Yes, I have it working 100% for powerscale image

but nothng for powerstore

image

donatwork commented 1 year ago

How did you perform the installation? Via Helm or Operator? If Operator please see the bottom of https://dell.github.io/csm-docs/docs/deployment/csmoperator/modules/observability/

Also check that all pods are running. It's difficult to trace without some logs. Are you able to query the otel collector for PowerStore metrics?

donatwork commented 1 year ago

Also what does kubectl -n karavi get pods show.

DTW2023 commented 1 year ago

there isn't a kavarai namespace? There is a dell-observability-csm name space which has

kubectl -n dell-observability-csm get pods

NAME READY STATUS RESTARTS AGE grafana-7c84fd97bc-fqx94 3/3 Running 0 2d22h karavi-metrics-powermax-74d67bc6bf-f7vbm 0/1 ContainerCreating 0 2d22h karavi-metrics-powerscale-697dd98f88-b582d 1/1 Running 0 2d22h karavi-metrics-powerstore-6589dd5494-vjxcg 1/1 Running 0 2d22h karavi-observability-cert-manager-684d8968d9-xsb4s 1/1 Running 0 2d22h karavi-observability-cert-manager-cainjector-68778db596-c58rv 1/1 Running 0 2d22h karavi-observability-cert-manager-webhook-69c4f48d7d-r964d 1/1 Running 0 2d22h karavi-topology-7bc84b666b-ww95k 1/1 Running 0 2d22h otel-collector-77b8956946-5mxqq 2/2 Running 0 2d22h prometheus-kube-state-metrics-7f6769f7c6-2gq6z 1/1 Running 0 2d22h prometheus-prometheus-node-exporter-cr6z6 1/1 Running 0 2d22h prometheus-prometheus-pushgateway-684dc6674-tnt5l 1/1 Running 0 2d22h prometheus-server-db986cf98-29882 1/1 Running 0 2d22h

donatwork commented 1 year ago

You can check the pogs logs to see if there are any problems. The karavi-metrics-powerstore, karavi-topology, and the powerstore driver logs. Is it possible to send the pods logs? On each worker node in /var/log/containers. You can send that via a private channel. I see the username of DTW2023, are you part of Dell? It can be much faster to troubleshoot if I can look at the system.

donatwork commented 1 year ago

Closing issue after discussing with submitter. We will look into this use case.