Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.33k stars 1.05k forks source link

Graylog is unable to determine opensearch disk space usage #19226

Open user29835461 opened 4 months ago

user29835461 commented 4 months ago

Expected Behavior

No errors in the logs.

Current Behavior

Constant spam in the logs:

2024-05-02T15:32:20.247+03:00 INFO  [ClusterAdapterOS2] _cat/nodes API has returned 1 nodes without disk statistics:
2024-05-02T15:32:20.248+03:00 INFO  [ClusterAdapterOS2] NodeResponse{id=b7SqnU8fTsm01QJUC9pUUw, name=data1, role=dimr, host=null, ip=xx.xx.xx.xx, diskUsed=null, diskTotal=null, diskUsedPercent=null, fileDescriptorMax=null}

Possible Solution

Just a guess, but has Opensearch changed the way API answers slightly, and the driver is unable to handle that?

To me the referenced API seems to respond with correct information:

$ curl -k -X GET "https://admin:password@server:9200/_cat/nodes?pretty"
10.xx.xx.xx 59 95  2  0.04  0.41  0.68 imr  ingest,master,remote_cluster_client               * witness
10.xx.xx.xx 39 73 99 17.88 15.62 14.77 dimr cluster_manager,data,ingest,remote_cluster_client - data1
10.xx.xx.xx 67 97 98 13.04 13.22 14.38 dimr cluster_manager,data,ingest,remote_cluster_client - data2

Steps to Reproduce (for bugs)

It happens all the time on its own.

Context

I have no idea if this breaks something. I noticed this when I investigated and other unrelated issue. I have no idea how to determine for how long this has been going on. The logs have roll over too fast in this environment.

Your Environment

kmerz commented 4 months ago

Same as https://github.com/Graylog2/graylog2-server/issues/19227#issuecomment-2090491818 probably related to the usage of OpenSearch 2.13.