search

Graylog2 / docker-compose

A set of Docker Compose files that allow you to quickly spin up a Graylog instance for testing or demo purposes.
Apache License 2.0
357 stars 134 forks source link

Cannot connect to data node #55

Closed jkobus closed 5 months ago

jkobus commented 5 months ago

Hello,

Im doing everything as per instructions, using the open core image, unfortunately as soon as I provision certificates in the web ui, everything stops working. I was looking everywhere, but it seems that Im unable to solve this on my own.

Any help will be appreciated.

app-graylog-1   | 2024-01-11 06:10:48,418 INFO : org.graylog2.security.CustomCAX509TrustManager - CA changed, refreshing trust manager
app-graylog-1   | 2024-01-11 06:10:48,423 INFO : org.graylog2.security.CustomCAX509TrustManager - CA changed, refreshing trust manager
app-datanode-1  | 2024-01-11T06:10:48.606Z INFO  [CustomCAX509TrustManager] CA changed, refreshing trust manager
app-datanode-1  | 2024-01-11T06:10:48.618Z INFO  [CustomCAX509TrustManager] CA changed, refreshing trust manager
app-datanode-1  | 2024-01-11T06:11:03.103Z INFO  [DataNodeConfigurationPeriodical] created CSR for this node
app-graylog-1   | 2024-01-11 06:11:06,289 INFO : org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical - Starting connectivity check with node 6d419d0b-1572-4e16-8135-f84b63b7be5e, silencing error messages for 60 seconds.
app-datanode-1  | 2024-01-11T06:11:06.609Z INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following alternative names: localhost, 127.0.0.1, 0:0:0:0:0:0:0:1, datanode, 0:0:0:0:0:0:0:1, 172.21.0.2
app-datanode-1  | 2024-01-11T06:11:06.657Z INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following alternative names: localhost, 127.0.0.1, 0:0:0:0:0:0:0:1, datanode, 0:0:0:0:0:0:0:1, 172.21.0.2
app-datanode-1  | 2024-01-11T06:11:06.763Z INFO  [TruststoreCreator] Adding certificate transport-chain-CA-root to the truststore
app-datanode-1  | 2024-01-11T06:11:06.770Z INFO  [TruststoreCreator] Adding certificate http-chain-CA-root to the truststore
app-datanode-1  | 2024-01-11T06:11:07.232Z INFO  [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.10.0-linux-x64/bin/opensearch
app-datanode-1  | 2024-01-11T06:11:08.538Z INFO  [JerseyService] Server configuration changed, restarting Data node REST API to apply security changes
app-datanode-1  | 2024-01-11T06:11:08.540Z INFO  [JerseyService] Shutting down HTTP listener at <0.0.0.0:8999>
app-datanode-1  | 2024-01-11T06:11:08.606Z INFO  [NetworkListener] Stopped listener bound to [0.0.0.0:8999]
app-datanode-1  | 2024-01-11T06:11:10.187Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:8999]
app-datanode-1  | 2024-01-11T06:11:10.199Z INFO  [HttpServer] [HttpServer-1] Started.
app-datanode-1  | 2024-01-11T06:11:10.202Z INFO  [JerseyService] Started REST API at <0.0.0.0:8999>
app-datanode-1  | 2024-01-11T06:11:10.370Z WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 90 unavailable. Cause: Connection refused
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:18.697+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.21.0.4:55862","uuid":"bec002f7-fe6d-46ba-a783-e66abf62abbc","connectionId":17,"connectionCount":11}}
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:18.702+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn17","msg":"client metadata","attr":{"remote":"172.21.0.4:55862","client":"conn17","doc":{"driver":{"name":"mongo-java-driver|legacy","version":"4.8.1"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"5.15.0-91-generic"},"platform":"Java/Eclipse Adoptium/17.0.9+9"}}}
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:18.727+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.21.0.4:55876","uuid":"e7157785-0afe-441a-850f-35e059f49486","connectionId":18,"connectionCount":12}}
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:18.738+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn18","msg":"client metadata","attr":{"remote":"172.21.0.4:55876","client":"conn18","doc":{"driver":{"name":"mongo-java-driver|legacy","version":"4.8.1"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"5.15.0-91-generic"},"platform":"Java/Eclipse Adoptium/17.0.9+9"}}}
app-datanode-1  | 2024-01-11T06:11:18.939Z WARN  [OpensearchProcessImpl] Opensearch process failed
app-datanode-1  | org.apache.commons.exec.ExecuteException: Process exited with an error: 137 (Exit value: 137)
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) ~[commons-exec-1.3.jar:1.3]
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) ~[commons-exec-1.3.jar:1.3]
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) [commons-exec-1.3.jar:1.3]
app-datanode-1  |   at java.lang.Thread.run(Unknown Source) [?:?]
app-datanode-1  | 2024-01-11T06:11:19.040Z INFO  [ProcessWatchdog] Detected terminated process, restarting. Attempt #1
app-datanode-1  | 2024-01-11T06:11:21.829Z INFO  [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.10.0-linux-x64/bin/opensearch
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:22.564+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1704953482:564704][1:0x7fb86738a700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 212, snapshot max: 212 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 1"}}
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:29.683+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn9","msg":"Slow query","attr":{"type":"command","ns":"graylog.cluster_events","command":{"find":"cluster_events","filter":{"consumers":{"$nin":["6d419d0b-1572-4e16-8135-f84b63b7be5e"]}},"sort":{"timestamp":1},"$db":"graylog","lsid":{"id":{"$uuid":"ab0775c9-4402-4a03-bef5-cc40966fe0cb"}}},"planSummary":"IXSCAN { timestamp: 1, producer: 1, consumers: 1 }","keysExamined":4,"docsExamined":2,"cursorExhausted":true,"numYields":1,"nreturned":0,"queryHash":"73F90509","planCacheKey":"918C06AD","reslen":111,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":2}},"Global":{"acquireCount":{"r":2}},"Mutex":{"acquireCount":{"r":1}}},"storage":{},"remote":"172.21.0.2:36678","protocol":"op_msg","durationMillis":103}}
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:29.683+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn18","msg":"Slow query","attr":{"type":"command","ns":"graylog.cluster_events","command":{"find":"cluster_events","filter":{"consumers":{"$nin":["347e3919-7eaa-4e3d-8179-e750d141812b"]}},"sort":{"timestamp":1},"$db":"graylog","lsid":{"id":{"$uuid":"42961960-0d35-4263-8868-9a198299440b"}}},"planSummary":"IXSCAN { timestamp: 1, producer: 1, consumers: 1 }","keysExamined":4,"docsExamined":2,"cursorExhausted":true,"numYields":1,"nreturned":0,"queryHash":"73F90509","planCacheKey":"918C06AD","reslen":111,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":2}},"Global":{"acquireCount":{"r":2}},"Mutex":{"acquireCount":{"r":1}}},"storage":{},"remote":"172.21.0.4:55876","protocol":"op_msg","durationMillis":106}}
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:29.683+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn15","msg":"Slow query","attr":{"type":"command","ns":"graylog.datanode_provisioning_config","command":{"find":"datanode_provisioning_config","filter":{"node_id":"6d419d0b-1572-4e16-8135-f84b63b7be5e"},"$db":"graylog","lsid":{"id":{"$uuid":"930baead-12da-4dfd-a299-5e50f5d7b283"}}},"planSummary":"IXSCAN { node_id: 1 }","keysExamined":1,"docsExamined":1,"cursorExhausted":true,"numYields":1,"nreturned":1,"queryHash":"5DE65DA6","planCacheKey":"752E0E7C","reslen":3329,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":2}},"Global":{"acquireCount":{"r":2}},"Mutex":{"acquireCount":{"r":1}}},"storage":{},"remote":"172.21.0.2:52574","protocol":"op_msg","durationMillis":107}}
app-datanode-1  | 2024-01-11T06:11:29.748Z WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 239 unavailable. Cause: Connection refused
app-datanode-1  | 2024-01-11T06:11:29.896Z WARN  [OpensearchProcessImpl] Opensearch process failed
app-datanode-1  | org.apache.commons.exec.ExecuteException: Process exited with an error: 137 (Exit value: 137)
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) ~[commons-exec-1.3.jar:1.3]
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) ~[commons-exec-1.3.jar:1.3]
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) [commons-exec-1.3.jar:1.3]
app-datanode-1  |   at java.lang.Thread.run(Unknown Source) [?:?]
app-datanode-1  | 2024-01-11T06:11:29.900Z INFO  [ProcessWatchdog] Detected terminated process, restarting. Attempt #2
app-datanode-1  | 2024-01-11T06:11:30.033Z INFO  [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.10.0-linux-x64/bin/opensearch
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:40.320+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.21.0.4:58522","uuid":"e18fe400-75c0-44ac-8fdf-2032d833c771","connectionId":19,"connectionCount":13}}
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:40.349+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn19","msg":"client metadata","attr":{"remote":"172.21.0.4:58522","client":"conn19","doc":{"driver":{"name":"mongo-java-driver|legacy","version":"4.8.1"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"5.15.0-91-generic"},"platform":"Java/Eclipse Adoptium/17.0.9+9"}}}
app-datanode-1  | 2024-01-11T06:11:40.375Z WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 385 unavailable. Cause: Connection refused
app-datanode-1  | 2024-01-11T06:11:40.583Z WARN  [OpensearchProcessImpl] Opensearch process failed
app-datanode-1  | org.apache.commons.exec.ExecuteException: Process exited with an error: 137 (Exit value: 137)
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) ~[commons-exec-1.3.jar:1.3]
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) ~[commons-exec-1.3.jar:1.3]
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) [commons-exec-1.3.jar:1.3]
app-datanode-1  |   at java.lang.Thread.run(Unknown Source) [?:?]
app-datanode-1  | 2024-01-11T06:11:40.586Z INFO  [ProcessWatchdog] Detected terminated process, restarting. Attempt #3
app-datanode-1  | 2024-01-11T06:11:40.721Z INFO  [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.10.0-linux-x64/bin/opensearch
app-datanode-1  | 2024-01-11T06:11:50.439Z WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 531 unavailable. Cause: Connection refused
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:50.560+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn12","msg":"Slow query","attr":{"type":"command","ns":"graylog.cluster_events","command":{"find":"cluster_events","filter":{"consumers":{"$nin":["347e3919-7eaa-4e3d-8179-e750d141812b"]}},"sort":{"timestamp":1},"$db":"graylog","lsid":{"id":{"$uuid":"653be3cf-9b76-4aa6-bd8b-e1970bef9452"}}},"planSummary":"IXSCAN { timestamp: 1, producer: 1, consumers: 1 }","keysExamined":4,"docsExamined":2,"cursorExhausted":true,"numYields":0,"nreturned":0,"queryHash":"73F90509","planCacheKey":"918C06AD","reslen":111,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":1}},"Global":{"acquireCount":{"r":1}},"Mutex":{"acquireCount":{"r":1}}},"storage":{},"remote":"172.21.0.4:40220","protocol":"op_msg","durationMillis":3379}}
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:50.587+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.21.0.4:46544","uuid":"948d7e68-8a7d-4432-bf93-685b205503a3","connectionId":20,"connectionCount":14}}
app-mongodb-1   | {"t":{"$date":"2024-01-11T06:11:50.598+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn20","msg":"client metadata","attr":{"remote":"172.21.0.4:46544","client":"conn20","doc":{"driver":{"name":"mongo-java-driver|legacy","version":"4.8.1"},"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"5.15.0-91-generic"},"platform":"Java/Eclipse Adoptium/17.0.9+9"}}}
app-datanode-1  | 2024-01-11T06:11:50.703Z WARN  [OpensearchProcessImpl] Opensearch process failed
app-datanode-1  | org.apache.commons.exec.ExecuteException: Process exited with an error: 137 (Exit value: 137)
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) ~[commons-exec-1.3.jar:1.3]
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) ~[commons-exec-1.3.jar:1.3]
app-datanode-1  |   at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) [commons-exec-1.3.jar:1.3]
app-datanode-1  |   at java.lang.Thread.run(Unknown Source) [?:?]
app-datanode-1  | 2024-01-11T06:11:50.711Z WARN  [ProcessWatchdog] Process watchdog terminated after too many restart attempts
jkobus commented 5 months ago

Ok it seems that it was the same issue (https://github.com/Graylog2/docker-compose/issues/46), not enough RAM :)