search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.24k stars 1.05k forks source link

Possible mismatch between endpoints elasticsearch / opensearch #13515

Closed byalexandrepedrosa closed 1 year ago

byalexandrepedrosa commented 1 year ago

Fresh install in LXD container can't get it working using Opensearch, works with Elasticsearch 7.10.2.

Expected Behavior

Work using Opensearch since Elasticsearch 7 will no longer receive updates.

Current Behavior

I tested using OpenJDK 8 to 18 and Oracle 8.

Initialization always loops on the line:

2022-09-20T12:34:09.735-03:00 WARN  [ServerBootstrap] Exception while running migrations
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to retrieve cluster information

Service logs, configurations, and environment variables:

Error from graylog.log

2022-09-19T13:27:10.048-03:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-09-19T13:27:10.062-03:00 INFO  [ServerBootstrap] Graylog server 4.3.7+05bccc7 starting up
2022-09-19T13:27:10.067-03:00 INFO  [ServerBootstrap] JRE: Private Build 18.0.2-ea on Linux 5.15.0-47-generic
2022-09-19T13:27:10.070-03:00 INFO  [ServerBootstrap] Deployment: deb
2022-09-19T13:27:10.071-03:00 INFO  [ServerBootstrap] OS: Ubuntu 22.04.1 LTS (jammy)
2022-09-19T13:27:10.071-03:00 INFO  [ServerBootstrap] Arch: amd64
2022-09-19T13:27:11.101-03:00 INFO  [ServerBootstrap] Running 62 migrations...
2022-09-19T13:27:12.817-03:00 WARN  [ServerBootstrap] Exception while running migrations
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to retrieve cluster information
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:151) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:111) ~[?:?]
        at org.graylog.storage.elasticsearch7.PlainJsonApi.perform(PlainJsonApi.java:38) ~[?:?]
        at org.graylog.storage.elasticsearch7.NodeAdapterES7.version(NodeAdapterES7.java:41) ~[?:?]
        at org.graylog2.indexer.cluster.Node.getVersion(Node.java:33) ~[graylog.jar:?]
        at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.getReopenedIndices(V20170607164210_MigrateReopenedIndicesToAliases.java:87) ~[graylog.jar:?]
        at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.getReopenedIndices(V20170607164210_MigrateReopenedIndicesToAliases.java:137) ~[graylog.jar:?]
        at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.lambda$upgrade$0(V20170607164210_MigrateReopenedIndicesToAliases.java:81) ~[graylog.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
        at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.upgrade(V20170607164210_MigrateReopenedIndicesToAliases.java:83) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.lambda$runMigrations$0(ServerBootstrap.java:264) ~[graylog.jar:?]
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422) ~[graylog.jar:?]
        at com.google.common.collect.RegularImmutableSortedSet.forEach(RegularImmutableSortedSet.java:88) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.runMigrations(ServerBootstrap.java:261) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:187) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:311) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
Caused by: java.net.ConnectException: Connection refused
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:849) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:259) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
        at org.graylog.storage.elasticsearch7.PlainJsonApi.lambda$perform$0(PlainJsonApi.java:40) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:109) ~[?:?]
        ... 23 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2022-09-19T13:27:27.161-03:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2022-09-19T13:27:29.119-03:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.3.7 [org.graylog.aws.AWSPlugin]
2022-09-19T13:27:29.122-03:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integrations 4.3.7 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2022-09-19T13:27:29.124-03:00 INFO  [CmdLineTool] Loaded plugin: Integrations 4.3.7 [org.graylog.integrations.IntegrationsPlugin]
2022-09-19T13:27:29.126-03:00 INFO  [CmdLineTool] Loaded plugin: Collector 4.3.7 [org.graylog.plugins.collector.CollectorPlugin]
2022-09-19T13:27:29.128-03:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise 4.3.7 [org.graylog.plugins.enterprise.EnterprisePlugin]
2022-09-19T13:27:29.133-03:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES6 Support) 4.3.7 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es6.EnterpriseES6Plugin]
2022-09-19T13:27:29.135-03:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES7 Support) 4.3.7 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es7.EnterpriseES7Plugin]
2022-09-19T13:27:29.137-03:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.3.7 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-09-19T13:27:29.140-03:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.3.7+05bccc7 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2022-09-19T13:27:29.141-03:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.3.7+05bccc7 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2022-09-19T13:27:29.214-03:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2022-09-19T13:27:30.400-03:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-09-19T13:27:30.500-03:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-09-19T13:27:30.579-03:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:1687}] to localhost:27017
2022-09-19T13:27:30.655-03:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=5087934}
2022-09-19T13:27:30.680-03:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:1688}] to localhost:27017
2022-09-19T13:27:30.733-03:00 INFO  [connection] Closed connection [connectionId{localValue:2, serverValue:1688}] to localhost:27017 because the pool has been closed.
2022-09-19T13:27:30.736-03:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 4.0.28
2022-09-19T13:27:31.364-03:00 INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <OpenSearch:1.3.4>
2022-09-19T13:27:31.552-03:00 INFO  [Version] HV000001: Hibernate Validator null
2022-09-19T13:27:39.826-03:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-09-19T13:27:39.844-03:00 INFO  [NodeId] Node ID: d236b99f-a6d7-4815-a1fc-75e9392550c2
2022-09-19T13:27:40.242-03:00 INFO  [LogManager] Loading logs.
2022-09-19T13:27:40.296-03:00 WARN  [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index...
2022-09-19T13:27:40.392-03:00 INFO  [LogManager] Logs loading complete.
2022-09-19T13:27:40.397-03:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2022-09-19T13:27:40.403-03:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-09-19T13:27:40.409-03:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-09-19T13:27:40.433-03:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:1689}] to localhost:27017
2022-09-19T13:27:40.435-03:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1655181}
2022-09-19T13:27:40.445-03:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:1690}] to localhost:27017
2022-09-19T13:27:40.644-03:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2022-09-19T13:27:41.072-03:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running OpenSearch:1.3.4
2022-09-19T13:27:42.295-03:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-09-19T13:27:42.428-03:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-09-19T13:27:42.444-03:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:1691}] to localhost:27017
2022-09-19T13:27:42.558-03:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:1692}] to localhost:27017
2022-09-19T13:27:42.623-03:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:1693}] to localhost:27017
2022-09-19T13:27:42.709-03:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:1694}] to localhost:27017
2022-09-19T13:27:42.876-03:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:1695}] to localhost:27017
2022-09-19T13:27:49.243-03:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-09-19T13:27:49.248-03:00 INFO  [ServerBootstrap] Graylog server 4.3.7+05bccc7 starting up
2022-09-19T13:27:49.257-03:00 INFO  [ServerBootstrap] JRE: Private Build 18.0.2-ea on Linux 5.15.0-47-generic
2022-09-19T13:27:49.259-03:00 INFO  [ServerBootstrap] Deployment: deb
2022-09-19T13:27:49.261-03:00 INFO  [ServerBootstrap] OS: Ubuntu 22.04.1 LTS (jammy)
2022-09-19T13:27:49.263-03:00 INFO  [ServerBootstrap] Arch: amd64
2022-09-19T13:27:49.439-03:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:1696}] to localhost:27017
2022-09-19T13:27:50.284-03:00 INFO  [ServerBootstrap] Running 62 migrations...
2022-09-19T13:27:52.004-03:00 WARN  [ServerBootstrap] Exception while running migrations
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to retrieve cluster information
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:151) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:111) ~[?:?]
        at org.graylog.storage.elasticsearch7.PlainJsonApi.perform(PlainJsonApi.java:38) ~[?:?]
        at org.graylog.storage.elasticsearch7.NodeAdapterES7.version(NodeAdapterES7.java:41) ~[?:?]
        at org.graylog2.indexer.cluster.Node.getVersion(Node.java:33) ~[graylog.jar:?]
        at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.getReopenedIndices(V20170607164210_MigrateReopenedIndicesToAliases.java:87) ~[graylog.jar:?]
        at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.getReopenedIndices(V20170607164210_MigrateReopenedIndicesToAliases.java:137) ~[graylog.jar:?]
        at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.lambda$upgrade$0(V20170607164210_MigrateReopenedIndicesToAliases.java:81) ~[graylog.jar:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
        at org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases.upgrade(V20170607164210_MigrateReopenedIndicesToAliases.java:83) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.lambda$runMigrations$0(ServerBootstrap.java:264) ~[graylog.jar:?]
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422) ~[graylog.jar:?]
        at com.google.common.collect.RegularImmutableSortedSet.forEach(RegularImmutableSortedSet.java:88) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.runMigrations(ServerBootstrap.java:261) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:187) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:311) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
Caused by: java.net.ConnectException: Connection refused
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:849) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:259) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
        at org.graylog.storage.elasticsearch7.PlainJsonApi.lambda$perform$0(PlainJsonApi.java:40) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:109) ~[?:?]
        ... 23 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Graylog server.conf:

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = <REMOVED>
root_password_sha2 = <REMOVED>
root_email = <REMOVED>
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.0.1.4:9000
http_publish_uri = http://graylog.mydomain.com/
http_external_uri = http://graylog.mydomain.com/
http_enable_cors = false
http_enable_tls = true
http_tls_cert_file = /etc/graylog/server/graylog.mydomain.com.crt
http_tls_key_file = /etc/graylog/server/graylog.mydomain.com.key
elasticsearch_hosts = https://admin:admin@graylog.mydomain.com:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Mongodb mongod.conf:

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

net:
  port: 27017
  bindIp: 127.0.0.1

processManagement:
  timeZoneInfo: /usr/share/zoneinfo

Opensearch opensearch.yml

cluster.name: graylog
action.auto_create_index: false
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 10.0.1.4
http.port: 9200
cluster.initial_master_nodes: ["graylog"]

plugins.security.ssl.transport.pemcert_filepath: graylog.mydomain.com.crt
plugins.security.ssl.transport.pemkey_filepath: graylog.mydomain.com.key
plugins.security.ssl.transport.pemtrustedcas_filepath: ca-certificates.crt
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.ssl.transport.enabled_protocols:
  - "TLSv1.3"
  - "TLSv1.2"
plugins.security.ssl.transport.enabled_ciphers:
  - "TLS_AES_256_GCM_SHA384"
  - "TLS_CHACHA20_POLY1305_SHA256"
  - "ECDHE-RSA-AES256-GCM-SHA384"
  - "ECDHE-RSA-CHACHA20-POLY1305"
  - "DHE-RSA-AES256-GCM-SHA384"
  - "ECDHE-RSA-AES128-GCM-SHA256"
  - "ECDHE-ECDSA-AES256-GCM-SHA384"
  - "ECDHE-ECDSA-CHACHA20-POLY1305"
  - "ECDHE-ECDSA-AES128-GCM-SHA256"
  - "DHE-RSA-AES128-GCM-SHA256"
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: graylog.mydomain.com-full.crt
plugins.security.ssl.http.pemkey_filepath: graylog.mydomain.com.key
plugins.security.ssl.http.pemtrustedcas_filepath: ca-certificates.crt
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.3"
  - "TLSv1.2"
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_AES_256_GCM_SHA384"
  - "TLS_CHACHA20_POLY1305_SHA256"
  - "ECDHE-RSA-AES256-GCM-SHA384"
  - "ECDHE-RSA-CHACHA20-POLY1305"
  - "DHE-RSA-AES256-GCM-SHA384"
  - "ECDHE-RSA-AES128-GCM-SHA256"
  - "ECDHE-ECDSA-AES256-GCM-SHA384"
  - "ECDHE-ECDSA-CHACHA20-POLY1305"
  - "ECDHE-ECDSA-AES128-GCM-SHA256"
  - "DHE-RSA-AES128-GCM-SHA256"
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=graylog.mydomain.com,OU=mydomain.com,O=mydomain

Tables at MongoDB are created using the following versions:

Example of table list:

access_tokens
alarmcallbackconfigurations
alarmcallbackhistory
alerts
archive_backends
archive_catalog
audit_log
cluster_config
cluster_events
collectors
content_packs
content_packs_installations
event_processor_state
forwarders
grants
index_failures
index_field_types
index_ranges
index_sets
licenses
lut_caches
lut_data_adapters
lut_tables
notifications
opensearch_anomaly_detectors
pipeline_processor_pipelines
pipeline_processor_pipelines_streams
pipeline_processor_rules
processing_status
roles
scheduler_triggers
searches
sessions
sidecar_configuration_variables
sidecars
streamrules
streams
system_messages
team_sync_backend_configs
teams
traffic
users

The version 4.3.1-1 stops at same error, but don't create tables on MongoDB.

Possible Solution

My guess is around endpoints don't be same so check fails on Opensearch because are using an inexistent endpoint and enter in loop for try n times.

Steps to Reproduce (for bugs)

Using a fresh install with the above config files would reproduce the problem (I tested using a second container and the same error occurs).

And follow the order:

  1. Be sure Graylog is not runing
  2. Start Mongodb
  3. Start Opensearch
  4. Start Graylog

Error should happen on the query about cluster.

Curiosity:

To make it run once with Opensearch:

  1. Be sure Graylog is not runing
  2. Start Mongodb
  3. Start Elasticsearch
  4. Start Graylog
  5. Stop Elasticsearch
  6. Start Opensearch

Once the Graylog has already passed the stage of verifying the cluster (even using single node) it can work without error with Opensearch (so I suspect it is a problem of querying an endpoint that does not exist in opensearch and only in the step of searching for cluster that occurs on every start/restart of Graylog).

Context

Financial companies with security certifications hardly accept solutions using abandoned packages or without updates, being practical in these to always opt for LTS Systems (read, it doesn't need to be edge, but it needs to have Long-term Support), in this way to be able to offer Graylog as a possible solution for Log management in these companies, it is necessary to comply with the policy they practice, in this case, since Elasticsearch will no longer receive updates when offering Graylog using Opensearch, which maintains updates, would be within these rules.

Your Environment

Ubuntu 22 comes with OpenSSL 3 and MongoDB are not ready for it, the workaround I have used is install the lib package from previous version libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb. Not sure if this would be an problem since bugs occours without encryption.

Have disable the cluster and used:

discovery.type: single-node

Does not work with Opensearch.

byalexandrepedrosa commented 1 year ago

It's not an bug, reading docs to hardening elasticsearch, have found the hint:

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/_discovery_configuration_check.html

If you do not need any discovery configuration, for instance if running a single-node cluster, set discovery.seed_hosts: [] to disable discovery and satisfy this bootstrap check.

Previously was using the setup from docs 7.10 (version in graylog manuals) steps said:

This bootstrap check ensures that discovery is not running with the default configuration. It can be satisfied by setting at least one of the following properties:

Then I have tested all 3 informing the instance as master, but not empty like docs on 7.17 says to.

To fix the infinite loop need use:

discovery.seed_hosts: []

Now its working, allowing have native encryption from tls and extra security options using opensearch, and with an package in active development (opensearch).