search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.4k stars 1.06k forks source link

graylog5 fails with opensearch 2.3.0 #13990

Closed ashish1099 closed 1 year ago

ashish1099 commented 1 year ago

Expected Behavior

The graylog should be working as it is as per the doc given here https://docs.graylog.org/docs/upgrading-to-opensearch-2x

Current Behavior

I can see this in the logs, the same error was seen with graylog 4.2 as well

2022-11-17 10:25:43,635 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #19). - {}
2022-11-17 10:25:44,068 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #19). - {}
2022-11-17 10:25:44,071 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #19). - {}
2022-11-17 10:26:13,814 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #20). - {}
2022-11-17 10:26:14,137 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #20). - {}
2022-11-17 10:26:14,141 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #20). - {}
2022-11-17 10:26:34,598 WARN    [LocalKafkaJournal] - Journal utilization (101.0%) has gone over 95%. - {}

Possible Solution

Steps to Reproduce (for bugs)

  1. Upgrade opensearch cluster to 2.x
  2. Upgrade mongodb to 5.x
  3. Upgrade the graylog to graylog 5.x
  4. graylogs fails to start

Context

Your Environment

Using graylog helm chart and deployed on k8s cluster (1.21.14)

janheise commented 1 year ago

@ashish1099 Hi, I just changed my local dev system from 2.2.0 to 2.3.0 - no problem. Also, if you have seen those error messages before, I doubt they are the root of the problem. Do you see anything else in the logs? How else is it failing for you? (no login? wrong results in searches etc.?)

ashish1099 commented 1 year ago

@janheise hey, thanks for the quick reply.

In this case the server never comes up and crashes

janheise commented 1 year ago

@ashish1099 Can you be more specific and post more info from the logs?

ashish1099 commented 1 year ago

@janheise

This is the only message that keeps getting repeated.

022-11-17 12:17:48,566 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #127). - {}
2022-11-17 12:17:48,616 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #127). - {}
2022-11-17 12:17:48,866 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #127). - {}

and when doing curl, we get 200 but no web page on the browser

root@ubuntu:~# curl http://100.115.3.44:9000/search -v
*   Trying 100.115.3.44:9000...
* Connected to 100.115.3.44 (100.115.3.44) port 9000 (#0)
> GET /search HTTP/1.1
> Host: 100.115.3.44:9000
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Frame-Options: DENY
< X-Graylog-Node-ID: 84f95b59-e98e-4fe3-8916-c84bdde44ed0
< X-UA-Compatible: IE=edge
< Content-Type: text/html
ashish1099 commented 1 year ago

and this is my opensearch cluster settings

root@ubuntu:~# curl -s -u admin:admin -X GET "http://opensearch-cluster-master:9200/_cluster/settings?pretty=true" 
{
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"
        }
      }
    },
    "compatibility" : {
      "override_main_response_version" : "true"
    },
    "plugins" : {
      "index_state_management" : {
        "template_migration" : {
          "control" : "-1"
        }
      }
    }
  },
  "transient" : { }
}
ashish1099 commented 1 year ago

pod logs since the start

Self IP is 100.115.3.24
[Try 2/3] Waiting for master node...
Current master is 
Self IP is 100.115.3.24
[Try 3/3] Waiting for master node...
Current master is 
Self IP is 100.115.3.24
[Try 4/3] Waiting for master node...
Launching graylog-0 as master
pod/graylog-0 labeled
Downloading Graylog Plugins...
Downloading https://github.com/KongZ/graylog-plugin-slack-notification/releases/download/v4.1.1/graylog-plugin-slack-notification-4.1.1.jar ...
Downloading https://github.com/KongZ/graylog-plugin-function-check-diff/releases/download/v1.0.1/graylog-plugin-function-check-diff-1.0.1.jar ...
Starting graylog
sh: 1: message_journal_dir: not found
Graylog Home /usr/share/graylog
Graylog Plugin Dir /usr/share/graylog/plugin
Graylog Elasticsearch Version 7
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-11-17 12:43:03,854 INFO    [ImmutableFeatureFlagsCollector] - Following feature flags are used: {default properties file=[search_filter=on]} - {}
2022-11-17 12:43:04,298 INFO    [CmdLineTool] - Loaded plugin: Compare actual result with expected result from query 1.0.1 [co.omise.graylog.plugins.checkdiff.CheckDiffFunctionPlugin] - {}
2022-11-17 12:43:04,300 INFO    [CmdLineTool] - Loaded plugin: Slack Notification Plugin 4.1.1 [com.kongz.graylog.plugins.slack.SlackNotificationPlugin] - {}
2022-11-17 12:43:04,301 INFO    [CmdLineTool] - Loaded plugin: AWS plugins 5.0.0-beta.5 [org.graylog.aws.AWSPlugin] - {}
2022-11-17 12:43:04,304 INFO    [CmdLineTool] - Loaded plugin: Integrations 5.0.0-beta.5 [org.graylog.integrations.IntegrationsPlugin] - {}
2022-11-17 12:43:04,305 INFO    [CmdLineTool] - Loaded plugin: Collector 5.0.0-beta.5 [org.graylog.plugins.collector.CollectorPlugin] - {}
2022-11-17 12:43:04,306 INFO    [CmdLineTool] - Loaded plugin: Threat Intelligence Plugin 5.0.0-beta.5 [org.graylog.plugins.threatintel.ThreatIntelPlugin] - {}
2022-11-17 12:43:04,306 INFO    [CmdLineTool] - Loaded plugin: Elasticsearch 7 Support 5.0.0-beta.5+04f3f36 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin] - {}
2022-11-17 12:43:04,307 INFO    [CmdLineTool] - Loaded plugin: OpenSearch 2 Support 5.0.0-beta.5+04f3f36 [org.graylog.storage.opensearch2.OpenSearch2Plugin] - {}
2022-11-17 12:43:04,325 INFO    [CmdLineTool] - Running with JVM arguments: -Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -XX:+UseContainerSupport -Dlog4j.configurationFile=/usr/share/graylog/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker - {}
2022-11-17 12:43:04,423 INFO    [cluster] - Adding discovered server mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 to client view of cluster - {}
2022-11-17 12:43:04,443 INFO    [cluster] - Adding discovered server mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 to client view of cluster - {}
2022-11-17 12:43:04,465 INFO    [client] - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.8.0-1041-aws"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog-user', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@108a46d6]}, clusterSettings={hosts=[mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='mongodb-replica-set', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null} - {}
2022-11-17 12:43:04,472 INFO    [client] - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.8.0-1041-aws"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog-user', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@108a46d6]}, clusterSettings={hosts=[mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='mongodb-replica-set', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null} - {}
2022-11-17 12:43:04,478 INFO    [connection] - Opened connection [connectionId{localValue:2, serverValue:3725}] to mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:04,479 INFO    [connection] - Opened connection [connectionId{localValue:4, serverValue:3731}] to mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:04,481 INFO    [connection] - Opened connection [connectionId{localValue:3, serverValue:3730}] to mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:04,481 INFO    [connection] - Opened connection [connectionId{localValue:1, serverValue:3726}] to mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:04,483 INFO    [cluster] - No server chosen by com.mongodb.client.internal.MongoClientDelegate$1@7c36db44 from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out - {}
2022-11-17 12:43:04,485 INFO    [cluster] - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=22856774, setName='mongodb-replica-set', canonicalAddress=mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, hosts=[mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017], passives=[], arbiters=[], primary='mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017', tagSet=TagSet{[]}, electionId=null, setVersion=2, topologyVersion=TopologyVersion{processId=6375fd923cecb46ad7b8ca1d, counter=8}, lastWriteDate=Thu Nov 17 12:42:54 UTC 2022, lastUpdateTimeNanos=705308975743636} - {}
2022-11-17 12:43:04,491 INFO    [cluster] - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=25317902, setName='mongodb-replica-set', canonicalAddress=mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, hosts=[mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017], passives=[], arbiters=[], primary='mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000009, setVersion=2, topologyVersion=TopologyVersion{processId=6375fe837f64cebf2468ea28, counter=6}, lastWriteDate=Thu Nov 17 12:42:54 UTC 2022, lastUpdateTimeNanos=705308978372922} - {}
2022-11-17 12:43:04,495 INFO    [cluster] - Setting max election id to 7fffffff0000000000000009 from replica set primary mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:04,495 INFO    [cluster] - Setting max set version to 2 from replica set primary mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:04,495 INFO    [cluster] - Discovered replica set primary mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:04,579 INFO    [connection] - Opened connection [connectionId{localValue:5, serverValue:3732}] to mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:04,613 INFO    [MongoDBPreflightCheck] - Connected to MongoDB version 5.0.13 - {}
2022-11-17 12:43:04,976 INFO    [SearchDbPreflightCheck] - Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2> - {}
2022-11-17 12:43:05,111 INFO    [Version] - HV000001: Hibernate Validator null - {}
2022-11-17 12:43:07,217 INFO    [InputBufferImpl] - Message journal is enabled. - {}
2022-11-17 12:43:07,229 INFO    [NodeId] - Node ID: 84f95b59-e98e-4fe3-8916-c84bdde44ed0 - {}
2022-11-17 12:43:07,350 INFO    [LogManager] - Loading logs. - {}
2022-11-17 12:43:07,354 INFO    [LogManager] - Logs loading complete. - {}
2022-11-17 12:43:07,367 INFO    [Log] - Completed load of log messagejournal-0 with log end offset 0 - {}
2022-11-17 12:43:07,377 INFO    [LogManager] - Created log for partition [messagejournal,0] in /usr/share/graylog/data/journal with properties {file.delete.delay.ms -> 60000, compact -> false, max.message.bytes -> 104857600, min.insync.replicas -> 1, segment.jitter.ms -> 0, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> true, retention.bytes -> 5368709120, delete.retention.ms -> 86400000, flush.ms -> 60000, segment.bytes -> 104857600, segment.ms -> 3600000, retention.ms -> 43200000, flush.messages -> 1000000, segment.index.bytes -> 1048576}. - {}
2022-11-17 12:43:07,391 INFO    [LocalKafkaJournal] - Initialized Kafka based journal at /usr/share/graylog/data/journal - {}
2022-11-17 12:43:07,395 INFO    [cluster] - Adding discovered server mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 to client view of cluster - {}
2022-11-17 12:43:07,397 INFO    [connection] - Opened connection [connectionId{localValue:6, serverValue:3730}] to mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:07,398 INFO    [cluster] - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1147271, setName='mongodb-replica-set', canonicalAddress=mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, hosts=[mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017], passives=[], arbiters=[], primary='mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017', tagSet=TagSet{[]}, electionId=null, setVersion=2, topologyVersion=TopologyVersion{processId=6375fd923cecb46ad7b8ca1d, counter=8}, lastWriteDate=Thu Nov 17 12:43:04 UTC 2022, lastUpdateTimeNanos=705311894986869} - {}
2022-11-17 12:43:07,403 INFO    [cluster] - Adding discovered server mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 to client view of cluster - {}
2022-11-17 12:43:07,405 INFO    [connection] - Opened connection [connectionId{localValue:7, serverValue:3731}] to mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:07,406 INFO    [client] - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.8.0-1041-aws"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog-user', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@108a46d6]}, clusterSettings={hosts=[mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='mongodb-replica-set', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null} - {}
2022-11-17 12:43:07,407 INFO    [connection] - Opened connection [connectionId{localValue:9, serverValue:3737}] to mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:07,407 INFO    [cluster] - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1342931, setName='mongodb-replica-set', canonicalAddress=mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, hosts=[mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017], passives=[], arbiters=[], primary='mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000009, setVersion=2, topologyVersion=TopologyVersion{processId=6375fe837f64cebf2468ea28, counter=6}, lastWriteDate=Thu Nov 17 12:43:04 UTC 2022, lastUpdateTimeNanos=705311904388287} - {}
2022-11-17 12:43:07,411 INFO    [connection] - Opened connection [connectionId{localValue:8, serverValue:3736}] to mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:07,415 INFO    [client] - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.8.0-1041-aws"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog-user', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@108a46d6]}, clusterSettings={hosts=[mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='mongodb-replica-set', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null} - {}
2022-11-17 12:43:07,415 INFO    [cluster] - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1147271, setName='mongodb-replica-set', canonicalAddress=mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, hosts=[mongodb-replica-set-0.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017], passives=[], arbiters=[], primary='mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017', tagSet=TagSet{[]}, electionId=null, setVersion=2, topologyVersion=TopologyVersion{processId=6375fd923cecb46ad7b8ca1d, counter=8}, lastWriteDate=Thu Nov 17 12:43:04 UTC 2022, lastUpdateTimeNanos=705311894986869}, ServerDescription{address=mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out - {}
2022-11-17 12:43:07,417 INFO    [cluster] - Setting max election id to 7fffffff0000000000000009 from replica set primary mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:07,418 INFO    [cluster] - Setting max set version to 2 from replica set primary mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:07,418 INFO    [cluster] - Discovered replica set primary mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:07,438 INFO    [connection] - Opened connection [connectionId{localValue:10, serverValue:3738}] to mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:07,530 INFO    [InputBufferImpl] - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers. - {}
2022-11-17 12:43:07,762 INFO    [ElasticsearchVersionProvider] - Elasticsearch version set to Elasticsearch:7.0.0 - disabling version probe. - {}
2022-11-17 12:43:08,191 INFO    [ProcessBuffer] - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>. - {}
2022-11-17 12:43:08,275 INFO    [connection] - Opened connection [connectionId{localValue:11, serverValue:3739}] to mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:08,279 INFO    [OutputBuffer] - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>. - {}
2022-11-17 12:43:08,731 INFO    [ServerBootstrap] - Graylog server 5.0.0-beta.5+04f3f36 starting up - {}
2022-11-17 12:43:08,731 INFO    [ServerBootstrap] - JRE: Eclipse Adoptium 17.0.5 on Linux 5.8.0-1041-aws - {}
2022-11-17 12:43:08,731 INFO    [ServerBootstrap] - Deployment: docker - {}
2022-11-17 12:43:08,731 INFO    [ServerBootstrap] - OS: Ubuntu 22.04.1 LTS (jammy) - {}
2022-11-17 12:43:08,731 INFO    [ServerBootstrap] - Arch: amd64 - {}
2022-11-17 12:43:08,808 INFO    [ServerBootstrap] - Running 49 migrations... - {}
2022-11-17 12:43:09,968 INFO    [PeriodicalsService] - Starting 26 periodicals ... - {}
2022-11-17 12:43:09,969 INFO    [Periodicals] - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s]. - {}
2022-11-17 12:43:09,985 INFO    [Periodicals] - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever. - {}
2022-11-17 12:43:10,067 INFO    [Periodicals] - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s]. - {}
2022-11-17 12:43:10,068 INFO    [LegacyDefaultStreamMigration] - Legacy default stream has no connections, no migration needed. - {}
2022-11-17 12:43:10,074 INFO    [Periodicals] - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s]. - {}
2022-11-17 12:43:10,079 INFO    [PeriodicalsService] - Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node. - {}
2022-11-17 12:43:10,079 INFO    [Periodicals] - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever. - {}
2022-11-17 12:43:10,080 INFO    [Periodicals] - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s]. - {}
2022-11-17 12:43:10,087 INFO    [Periodicals] - Starting [org.graylog2.periodical.IndexBlockCheck] periodical in [0s], polling every [30s]. - {}
2022-11-17 12:43:10,094 INFO    [Periodicals] - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s]. - {}
2022-11-17 12:43:10,108 INFO    [Periodicals] - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s]. - {}
2022-11-17 12:43:10,123 INFO    [Periodicals] - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s]. - {}
2022-11-17 12:43:10,133 INFO    [Periodicals] - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s]. - {}
2022-11-17 12:43:10,139 INFO    [Periodicals] - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s]. - {}
2022-11-17 12:43:10,146 INFO    [Periodicals] - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. - {}
2022-11-17 12:43:10,151 INFO    [Periodicals] - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s]. - {}
2022-11-17 12:43:10,175 INFO    [Periodicals] - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s]. - {}
2022-11-17 12:43:10,190 INFO    [Periodicals] - Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s]. - {}
2022-11-17 12:43:10,192 INFO    [Periodicals] - Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [1s]. - {}
2022-11-17 12:43:10,193 INFO    [connection] - Opened connection [connectionId{localValue:12, serverValue:3740}] to mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:10,211 INFO    [Periodicals] - Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s]. - {}
2022-11-17 12:43:10,215 INFO    [Periodicals] - Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s]. - {}
2022-11-17 12:43:10,217 INFO    [Periodicals] - Starting [org.graylog2.periodical.UserSessionTerminationPeriodical] periodical, running forever. - {}
2022-11-17 12:43:10,218 INFO    [Periodicals] - Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s]. - {}
2022-11-17 12:43:10,221 INFO    [Periodicals] - Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s]. - {}
2022-11-17 12:43:10,222 INFO    [Periodicals] - Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s]. - {}
2022-11-17 12:43:10,250 INFO    [Periodicals] - Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s]. - {}
2022-11-17 12:43:10,250 INFO    [Periodicals] - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s]. - {}
2022-11-17 12:43:10,359 INFO    [PrometheusExporterHTTPServer] - Exporting Prometheus metrics on <0.0.0.0:9833> via HTTP - {}
2022-11-17 12:43:10,882 WARN    [AuditEventModelProcessor] - REST endpoint not included in audit trail:   POST api/plugins/com.kongz.graylog.plugins.slack/action - {}
2022-11-17 12:43:11,847 INFO    [JerseyService] - Started REST API at <0.0.0.0:9000> - {}
2022-11-17 12:43:11,848 INFO    [ServiceManagerListener] - Services are healthy - {}
2022-11-17 12:43:11,848 INFO    [ServerBootstrap] - Services started, startup times in ms: {LocalKafkaMessageQueueWriter [RUNNING]=0, UrlWhitelistService [RUNNING]=0, UserSessionTerminationService [RUNNING]=0, FailureHandlingService [RUNNING]=1, InputSetupService [RUNNING]=4, LocalKafkaMessageQueueReader [RUNNING]=5, BufferSynchronizerService [RUNNING]=9, EtagService [RUNNING]=12, OutputSetupService [RUNNING]=12, JobSchedulerService [RUNNING]=14, GracefulShutdownService [RUNNING]=15, StreamCacheService [RUNNING]=19, GeoIpDbFileChangeMonitorService [RUNNING]=21, ConfigurationEtagService [RUNNING]=31, MongoDBProcessingStatusRecorderService [RUNNING]=52, LocalKafkaJournal [RUNNING]=62, LookupTableService [RUNNING]=70, PeriodicalsService [RUNNING]=282, PrometheusExporter [RUNNING]=324, JerseyService [RUNNING]=1879} - {}
2022-11-17 12:43:11,852 INFO    [InputSetupService] - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] - {}
2022-11-17 12:43:11,855 INFO    [ServerBootstrap] - Graylog server up and running. - {}
2022-11-17 12:43:11,930 INFO    [InputLauncher] - Launching input [Beats/Beats/615efc8875115e045b4ce600] - desired state is RUNNING - {}
2022-11-17 12:43:11,942 INFO    [InputStateListener] - Input [Beats/615efc8875115e045b4ce600] is now STARTING - {}
2022-11-17 12:43:12,021 INFO    [InputStateListener] - Input [Beats/615efc8875115e045b4ce600] is now RUNNING - {}
2022-11-17 12:43:28,702 INFO    [connection] - Opened connection [connectionId{localValue:13, serverValue:3747}] to mongodb-replica-set-1.mongodb-replica-set-svc.graylog.svc.cluster.local:27017 - {}
2022-11-17 12:43:29,971 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #1). - {}
2022-11-17 12:43:29,973 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #1). - {}
2022-11-17 12:43:29,972 ERROR   [Messages] - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #1). - {}
janheise commented 1 year ago

@ashish1099 do you have the ES7 compatibility setting in your graylog.conf? 2022-11-17 12:43:04,976 INFO [SearchDbPreflightCheck] - Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2> - {}

ashish1099 commented 1 year ago

this is what I have in graylog.conf

node_id_file = /usr/share/graylog/data/journal/node-id
root_username = admin
root_email = 
root_timezone = UTC

http_bind_address = 0.0.0.0:9000
http_external_uri = graylog.io/
elasticsearch_hosts = http://${GRAYLOG_ELASTICSEARCH_HOSTS}
allow_leading_wildcard_searches = true
allow_highlighting = true
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
# Do not change `message_journal_dir` location
message_journal_dir = /usr/share/graylog/data/journal
message_journal_max_size = 5gb
lb_recognition_period_seconds = 3
# Use a replica set instead of a single host
mongodb_uri = ${GRAYLOG_MONGODB_URI}
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
# Email transport
transport_email_enabled = false
transport_email_hostname = 
transport_email_port = 2587
transport_email_use_auth = true
transport_email_use_tls = true
transport_email_use_ssl = false
transport_email_auth_username = 
transport_email_auth_password = 
transport_email_subject_prefix = [graylog]
transport_email_from_email = 
gc_warning_threshold = 1s
transport_email_web_interface_url = graylog.io
content_packs_dir = /usr/share/graylog/data/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
prometheus_exporter_enabled = true
prometheus_exporter_bind_address = 0.0.0.0:9833
janheise commented 1 year ago

other environment variables you're setting?

just for a reference, my log states: org.graylog2.bootstrap.preflight.SearchDbPreflightCheck - Connected to (Elastic/Open)Search version <OpenSearch:2.3.0>

ashish1099 commented 1 year ago

@janheise do you have elasticsearch_version set in the graylog conf ?

janheise commented 1 year ago

no, my version is read from the searchserver. maybe elasticsearch_hosts = http://${GRAYLOG_ELASTICSEARCH_HOSTS} points to an incorrect server?

ashish1099 commented 1 year ago

looks like this is wrong

elasticsearch_hosts = http://http://admin:admin@opensearch-cluster-master:9200

let me get this fixed and test

ashish1099 commented 1 year ago

ok get this fixed as well, but still same error :(

janheise commented 1 year ago

Please check if your cluster really runs OpenSearch 2.3 - and not an old version, again, check also for other environment variables that you're setting. From the preflight check: you probably still run your OpenSearch 1.2.1 in ES7 compatibility mode or force your OpenSearch 2 into compatibility mode (don't even know if it still exists with 2.x)

KlavsKlavsen commented 1 year ago

it seems to be because we have set this in open search: compatibility.override_main_response_version: true

KlavsKlavsen commented 1 year ago

The odd thing is that we tested this - even with graylog 4.x - on a new blank install - and there it seems to work fine. We did not try to load in larger amounts of data in it - we merely checked system status - which was green.

janheise commented 1 year ago

@KlavsKlavsen so, is it working now and the issue can be closed?

ashish1099 commented 1 year ago

@janheise no it didnt worked, I got the log when graylog said its opensearch but ended up with same error message

graylog-server 2022-11-17 13:44:51,476 INFO    [SearchDbPreflightCheck] - Connected to (Elastic/Open)Search version <OpenSearch:2.3.0> - {}                                                                    │
janheise commented 1 year ago

@ashish1099 sorry, that it's still not working for you. A couple of things I want to ask you to check:

ashish1099 commented 1 year ago

@janheise I figured out the issue, it was http_external_uri which was missing https://