search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.31k stars 1.05k forks source link

IndexFieldTypePollerPeriodical throws occasionally ResponseException #19227

Open user29835461 opened 4 months ago

user29835461 commented 4 months ago

Expected Behavior

No errors in the logs.

Current Behavior

This error shows occasionally in the logs. It refers to the Graylog Events index set.

2024-05-02T15:18:01.048+03:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in Periodical
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchException: Couldn't get index list for index set <5d89b76d0a25285d26afd396>
        at org.graylog.storage.opensearch2.OpenSearchClient.exceptionFrom(OpenSearchClient.java:182) ~[?:?]
        at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:133) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.perform(CatApi.java:105) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.requestIndices(CatApi.java:101) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.indices(CatApi.java:76) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.indices(CatApi.java:71) ~[?:?]
        at org.graylog.storage.opensearch2.IndicesAdapterOS2.indices(IndicesAdapterOS2.java:414) ~[?:?]
        at org.graylog2.indexer.indices.Indices.getIndices(Indices.java:283) ~[graylog.jar:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePoller.poll(IndexFieldTypePoller.java:68) ~[graylog.jar:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$refreshFieldTypes$2(IndexFieldTypePollerPeriodical.java:155) ~[graylog.jar:?]
        at java.lang.Iterable.forEach(Unknown Source) ~[?:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.refreshFieldTypes(IndexFieldTypePollerPeriodical.java:144) ~[graylog.jar:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.doRun(IndexFieldTypePollerPeriodical.java:131) ~[graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [GET], host [https://data1:9200], URI [/_cat/indices/gl-events_*?format=json&h=index%2Cstatus&s=index%2Cstatus&expand_wildcards=all], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1328b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1328b"},"status":400}
        at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:375) ~[?:?]
        at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:345) ~[?:?]
        at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:320) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.lambda$perform$4(CatApi.java:108) ~[?:?]
        at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:131) ~[?:?]
        ... 18 more

Possible Solution

No idea. The storage driver for Opensearch does something wrong?

Steps to Reproduce (for bugs)

It happens on its own.

The API from Opensearch seems to give proper information:

$ curl -k -X GET "https://admin:passwowrd@data1:9200/_cat/indices/gl-events_*?format=json&h=index%2Cstatus&s=index%2Cstatus&expand_wildcards=all&pretty"
[
  {
    "index" : "gl-events_10",
    "status" : "open"
  },
  {
    "index" : "gl-events_11",
    "status" : "open"
  },
  {
    "index" : "gl-events_12",
    "status" : "open"
  },
  {
    "index" : "gl-events_3",
    "status" : "open"
  },
  {
    "index" : "gl-events_4",
    "status" : "open"
  },
  {
    "index" : "gl-events_5",
    "status" : "open"
  },
  {
    "index" : "gl-events_6",
    "status" : "open"
  },
  {
    "index" : "gl-events_7",
    "status" : "open"
  },
  {
    "index" : "gl-events_8",
    "status" : "open"
  },
  {
    "index" : "gl-events_9",
    "status" : "open"
  }
]

Context

I have no idea what this might cause.

Your Environment

dennisoelkers commented 4 months ago

Hey @user29835461!

Thanks for reporting this and providing helpful information! My current impression is that it might be related to OpenSearch 2.13 not being supported completely by the 2.9 client library included in 5.2.6. 6.0 will include a more recent client library that could potentially fix this issue. In the meantime I will check if I can reproduce the issue locally.