search

IBM / ibm-spectrum-scale-bridge-for-grafana

This tool allows the IBM Storage Scale users to perform performance monitoring for IBM Storage Scale devices using third-party applications such as Grafana or Prometheus software.
Apache License 2.0
30 stars 17 forks source link

No Metadata results received from the pmcollector #237

Open CuriousUser opened 2 days ago

CuriousUser commented 2 days ago

Hello, I’m trying to get grafana bridge up and running in a container (https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/wiki/Running-the-IBM-Spectrum-Scale-Performance-Monitoring-Bridge-in-a-docker-container). I managed to start it but getting “No Metadata results received from the pmcollector” message. I can’t find what is missing. Tried different options for starting container, but same message appears. Network port is opened in firewall. What could it be?

Logs from container:

Output from "podman logs prometheus-exporter"

Warning: You are running a Development version of the IBM Storage Scale bridge for Grafana.
It is recommended to use the latest released version, published on:
https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/releases

2024-09-02 10:38 - MainThread                               - INFO     -  *** IBM Storage Scale bridge for Grafana - Version: 8.0.1-dev ***
2024-09-02 10:40 - MainThread                               - ERROR    - QueryHandler: getTopology returns no data.
2024-09-02 10:40 - MainThread                               - WARNING  - No Metadata results received from the pmcollector. Start retry attempt 1 in 60s (MAX_ATTEMPTS_COUNT:3)
2024-09-02 10:43 - MainThread                               - ERROR    - QueryHandler: getTopology returns no data.
2024-09-02 10:43 - MainThread                               - WARNING  - No Metadata results received from the pmcollector. Start retry attempt 2 in 60s (MAX_ATTEMPTS_COUNT:3)
2024-09-02 10:47 - MainThread                               - ERROR    - QueryHandler: getTopology returns no data.
2024-09-02 10:47 - MainThread                               - WARNING  - No Metadata results received from the pmcollector. Start retry attempt 3 in 60s (MAX_ATTEMPTS_COUNT:3)
%s Server internal error occurred. Reason: Empty results received
2024-09-02 10:48 - MainThread                               - ERROR    - Metadata could not be retrieved. Check log file for more details, quitting

Bridge is running fine with same API key directly on EMS, just version is older:

Output from "python3 zimonGrafanaIntf.py --apiKeyValue xxxxx"

2024-08-30 15:00 - INFO     -  *** IBM Spectrum Scale bridge for Grafana - Version: 7.0.8 ***
2024-08-30 15:00 - INFO     - Successfully retrieved MetaData
2024-08-30 15:00 - INFO     - Received sensors:CPU, DiskFree, GPFSFilesystem, GPFSFilesystemAPI, GPFSNode, GPFSNodeAPI, GPFSRPCS, GPFSVFSX, GPFSWaiters, Load, Memory, Netstat, Network, TopProc, CTDBDBStats, CTDBStats, NFSIO, SMBGlobalStats, SMBStats, GPFSDiskCap, GPFSFileset, GPFSFilesetQuota, GPFSInodeCap, GPFSPool, GPFSPoolCap
2024-08-30 15:00 - INFO     - Initial cherryPy server engine start have been invoked. Python version: 3.6.8 (default, Jan 14 2022, 11:10:48)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-7)], cherryPy version: 18.6.1.
2024-08-30 15:00 - INFO     - server started
^C2024-08-30 15:00 - WARNING  - server stopped

Port 9250 is reachable to EMS:

Output from "nc -v EMS-IP 9250"
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connection refused.

Command to start bridge in container:

podman run -dt -p 9250:9250 --privileged -e "SERVER=EMS-IP" -e "BASICAUTHPASSW=xxx" -e "APIKEYVALUE=xxx" -e "PROMETHEUS=9250" -e "TLSKEYPATH=/etc/bridge_ssl/certs" -e "TLSKEYFILE=privkey.pem" -e "TLSCERTFILE=cert.pem" -v /tmp:/var/log/ibm_bridge_for_grafana -v /etc/bridge_ssl/certs:/etc/bridge_ssl/certs --mount type=bind,src=/opt/IBM/zimon/ZIMonSensors.cfg,target=/opt/IBM/zimon/ZIMonSensors.cfg,ro=true --pod new:my-bridge --name prometheus-exporter bridge_image:latest

Helene commented 2 days ago

@CuriousUser, please run "podman run -dt" with all parameters once again. After that, capture and paste here the output of 

# cat /tmp/zserver.log
CuriousUser commented 2 days ago

@Helene

Output of "cat /tmp/zserver.log"

2024-09-25 13:08:49,373 - bridgeLogger - MainThread                                              - INFO     -  *** IBM Storage Scale bridge for Grafana - Version: 8.0.1-dev ***
2024-09-25 13:08:49,373 - bridgeLogger - MainThread                                              - MOREINFO - zimonGrafanaItf invoked with parameters:
 prometheus=9250
rawCounters=True
protocol=http
enabled=True
username=monitor
server=EMS-IP
serverPort=9980
retryDelay=60
apiKeyName=scale_grafana
caCertPath=False
includeDiskData=False
logPath=/var/log/ibm_bridge_for_grafana
logLevel=10
logFile=zserver.log
tlsKeyPath=/etc/bridge_ssl/certs
tlsKeyFile=privkey.pem
tlsCertFile=cert.pem
2024-09-25 13:08:49,373 - bridgeLogger - MainThread                                              - DEBUG    - MMSDRFS file not found (/var/mmfs/gen/mmsdrfs), continue with trying to read pmsensors configuration from ZIMonSensors.cfg
2024-09-25 13:08:49,374 - bridgeLogger - MainThread                                              - DEBUG    - readSensorsConfig attempt to read /opt/IBM/zimon
2024-09-25 13:11:01,152 - bridgeLogger - MainThread                                              - MOREINFO - Perfmon RESTcall error __ Server responded: 503 Connection refused from server
2024-09-25 13:11:01,152 - bridgeLogger - MainThread                                              - ERROR    - QueryHandler: getTopology returns no data.
2024-09-25 13:11:01,153 - bridgeLogger - MainThread                                              - WARNING  - No Metadata results received from the pmcollector. Start retry attempt 1 in 60s (MAX_ATTEMPTS_COUNT:3)
2024-09-25 13:14:09,568 - bridgeLogger - MainThread                                              - MOREINFO - Perfmon RESTcall error __ Server responded: 503 Connection refused from server
2024-09-25 13:14:09,568 - bridgeLogger - MainThread                                              - ERROR    - QueryHandler: getTopology returns no data.
2024-09-25 13:14:09,569 - bridgeLogger - MainThread                                              - WARNING  - No Metadata results received from the pmcollector. Start retry attempt 2 in 60s (MAX_ATTEMPTS_COUNT:3)
2024-09-25 13:17:17,984 - bridgeLogger - MainThread                                              - MOREINFO - Perfmon RESTcall error __ Server responded: 503 Connection refused from server
2024-09-25 13:17:17,984 - bridgeLogger - MainThread                                              - ERROR    - QueryHandler: getTopology returns no data.
2024-09-25 13:17:17,984 - bridgeLogger - MainThread                                              - WARNING  - No Metadata results received from the pmcollector. Start retry attempt 3 in 60s (MAX_ATTEMPTS_COUNT:3)
2024-09-25 13:18:17,998 - bridgeLogger - MainThread                                              - ERROR    - Metadata could not be retrieved. Check log file for more details, quitting
Helene commented 2 days ago

It looks like the communication with pmcollector could not be established properly

 Server responded: 503 Connection refused from server

The bridge application does try to get data from pmcollector using 

server=EMS-IP
serverPort=9980

"server" - should be the reachable ip address of the pmcollector node port 9980 should be not blocked.

Are you using any Proxy or Firewall?

CuriousUser commented 1 day ago

Yes, there is firewall in between. I was not aware about this port ☹️ Once firewall is open I'll try to run again. Thank you!

Helene commented 1 day ago

Yes, there is firewall in between. I was not aware about this port ☹️ Once firewall is open I'll try to run again. Thank you!

Let me know if it works or you have still problems.