search

IBM / ibm-spectrum-scale-bridge-for-grafana

This tool allows the IBM Storage Scale users to perform performance monitoring for IBM Storage Scale devices using third-party applications such as Grafana or Prometheus software.
Apache License 2.0
32 stars 19 forks source link

How works Prometheus Integration ? #169

Open SckyzO opened 9 months ago

SckyzO commented 9 months ago

Hello team,

I see that IBm bridge for grafana support promtheus.

It is possible to explain how it is working (update readme maybe) ? And how to use it ?

And can you update your dockerfile to use Prometheus Exporter API ?

Regards,

Helene commented 9 months ago

Hi @SckyzO, Prometheus support is currently in the implementation phase. The Dockerfile update is also on the development todo list. As soon as the development is finished, I'll post a clear user guide on the wiki. If you want to try out the current code, say a beta version, I can provide you a short instructions on how to set it up.

Helene

SckyzO commented 9 months ago

Hello @Helene

Ok, I understand :) This will definitely be a good feature :)

Do you know when a final version with official Prometheus support will arrive?

I would like a "docker/podman" version even in beta. I have to implement it for a client and I would like to start working on the subject. I understood how it worked but in container mode it is not yet supported (I adapted the dockerfile but there is still work). The problem is that I am on a closed network, so the python pip offline part must be managed offline. I would therefore prefer to make it work with containers :)

Tom

Helene commented 9 months ago

Hi @SckyzO, I updated the Dockerfile. Now you should be able build and run grafana-bridge as prometheus-exporter in a container. Check PR #172 for some examples. The wiki article will be updated once implementation and testing are complete. Don't forget to create ssl certificates prior you start with pod deployment.

Your early feedback would be greatly appreciated.

SckyzO commented 8 months ago

Hello Helene. I can't run my container, I have many error with cherry and certificate :

For example : 

podman run -p 9250:9250 -e "SERVER=myserver" -e "APIKEYVALUE=XXXXX" -e "PROMETHEUS=9250" -e "TLSKEYPATH=/etc/bridge_ssl/certs" -e "TLSKEYFILE=privkey.pem" -e "TLSCERTFILE=cert.pem" -v /tmp/logs:/var/log/ibm_bridge_for_grafana -v /tmp/certs:/etc/bridge_ssl/certs --pod new:bridge_ibm-pod --name grafana_bridge docker.io/library/bridge_ibm:latest
[22/Feb/2024:14:20:23] ENGINE Error in HTTPServer.serve
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cheroot/server.py", line 1814, in serve
    self._connections.run(self.expiration_interval)
  File "/usr/local/lib/python3.9/site-packages/cheroot/connections.py", line 198, in run
    self._run(expiration_interval)
  File "/usr/local/lib/python3.9/site-packages/cheroot/connections.py", line 241, in _run
    new_conn = self._from_server_socket(self.server.socket)
  File "/usr/local/lib/python3.9/site-packages/cheroot/connections.py", line 294, in _from_server_socket
    s, ssl_env = self.server.ssl_adapter.wrap(s)
  File "/usr/local/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 270, in wrap
    s = self.context.wrap_socket(
  File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.9/ssl.py", line 1074, in _create
    self.do_handshake()
  File "/usr/lib64/python3.9/ssl.py", line 1343, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129)

And, if we use container with certificate, why do not add certificate directly in the container ? It is a localhost self-signed certificate Another bug, log folder haven't good right, logrotate can't work and if you mount local path with container log path (/var/log/ibm_bridge_for_grafana), container can't create log files. Workarround, local folder 777 mode and create empty log files with 777 mode. 😄

I continue to debug :) I don't know why, I started my container one time, but after kill and re-run container, it never restarted ...

Another question, I have to recover mmsdrfs, but in my case, you have to add hardware gradually to my ESS architecture, so I think that this file should be passed as an argument (mount a local path with this file I think) , because I will have to rebuild the container if my architecture evolves.

SckyzO commented 8 months ago

Juste for information, my container is started, i always have "ssl error" as said in my previons comment, but it working.

In log, I have many Warnings

Like this : 

[ . . . ]
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_read
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_bytes_written
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_disks
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_max_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_min_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_read_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_disk_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_rd
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_tot_queue_wait_wr
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-29 13:39 - CP Server Thread-12            - WARNING  - Received null values in all data points for gpfs_fs_write_ops

I don't know why and how to check this warnings ... If you have an idea :)

SckyzO commented 8 months ago

Oh, I found in code the list of all endpoints :)

https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/blob/master/source/prometheus_endpoints.json

Maybe you should update Prometheus Readme with this list :)

@Helene very good job pour this prometheus integration development

Helene commented 8 months ago

Hello @SckyzO , I'm very sorry for answering with delay to you. I was very busy with another project. Many thanks for your feedback.

The ssl exception you are seening is comming from CherryPy application. It is known issue. Finally it has been fixed 2 weeks ago but not published yet from CherryPy development team. You can follow on this https://github.com/cherrypy/cheroot/pull/518.

Workarround, local folder 777 mode and create empty log files with 777 mode. 😄

That is correct your local folder, where you mount the grafana-bridge logs, need to have write permissions, otherwise the log files could not be updated. Are you running grafana-bridge container on baremetal or on a kubernetes cluster. On kubernetes you can make the installing process much easier using deployment script.

Another question, I have to recover mmsdrfs, but in my case, you have to add hardware gradually to my ESS architecture, so I think that this file should be passed as an argument (mount a local path with this file I think) , because I will have to rebuild the container if my architecture evolves.

This is a good point. Mounting mmsdrfs/ZIMonsensors.cfg file would make sense. Please open for this a separate issue.

I'm a little bit worry about the messages

2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops
2024-02-27 09:28 - CP Server Thread-10            - WARNING  - Received null values in all data points for gpfs_fs_write_ops

Are you using the latest grafana-bridge code? Could you please paste me your settings in the config.ini for prometheus part. Something like this:

##################### Prometheus Exporter API Connection Defaults #############
# Port number the bridge listening on for Prometheus server https requests;
# ssl cert and key configuration required
prometheus = 9250

# Sensor counters deltas will be exported by default.
# Set True if you want export original sensor counters.
rawCounters = True

Also verify your prometheus config file prometheus.yaml has a correct job definition

  - job_name: 'GPFSFilesystem_cluster1'
    scrape_interval: 300s
    honor_timestamps: true
    metrics_path: '/metrics_gpfs_filesystem'
    scheme: https
    tls_config:
      cert_file: /etc/prometheus/certs/cluster1/cert.pem
      key_file: /etc/prometheus/certs/cluster1/privkey.pem
      insecure_skip_verify: true
    static_configs:
    - targets: ['9.152.x.x:9250']

Oh, I found in code the list of all endpoints :)

https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/blob/master/source/prometheus_endpoints.json

Maybe you should update Prometheus Readme with this list :)

@Helene very good job pour this prometheus integration development

Yes the endpoints need to be documented. I'm still working on documentation. Many thanks for the praise !