search

mrlesmithjr / graylog2

GNU General Public License v2.0
118 stars 57 forks source link

Graylog2-server will not launch after first reboot (CentOS only) #22

Closed Kev-Johnson closed 10 years ago

Kev-Johnson commented 10 years ago

Hi Larry - thanks for your script. I've been trying to figure this out for a while with little success: if I use the Ubuntu script all is great, however if I use the CentOS script the install appears to complete succesfully but graylog2-server fails to launch on reboot. Manually launching the server then checking tail of graylog2-server.log shows me

2014-07-06 07:32:05,969 INFO : org.graylog2.Main - Graylog2 0.20.4 starting up. (JRE: Oracle >Corporation 1.7.0_55 on Linux 2.6.32$ 2014-07-06 07:33:17,041 INFO : org.graylog2.plugin.system.NodeId - Node ID: d375e0eb-07ff->494e-bf51-f17ea5be758a 2014-07-06 07:34:07,254 INFO : org.graylog2.buffers.ProcessBuffer - Initialized ProcessBuffer with >ring size <1024> and wait stra$ 2014-07-06 07:34:07,271 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with >ring size <1024> and wait strate$ 2014-07-06 07:34:07,796 INFO : org.elasticsearch.node - [graylog2-server] version[0.90.10], >pid[6811], build[0a5781f/2014-01-10T1$ 2014-07-06 07:34:07,796 INFO : org.elasticsearch.node - [graylog2-server] initializing ... 2014-07-06 07:34:07,801 INFO : org.elasticsearch.plugins - [graylog2-server] loaded [], sites [] 2014-07-06 07:34:10,031 INFO : org.elasticsearch.node - [graylog2-server] initialized 2014-07-06 07:34:10,031 INFO : org.elasticsearch.node - [graylog2-server] starting ... 2014-07-06 07:34:10,092 INFO : org.elasticsearch.transport - [graylog2-server] bound_address >{inet[/0.0.0.0:9350]}, publish_addre$ 2014-07-06 07:34:13,096 WARN : org.elasticsearch.discovery - [graylog2-server] waited for 3s and >no initial state was set by the $ 2014-07-06 07:34:13,097 INFO : org.elasticsearch.discovery - [graylog2-server] >graylog2/BezylqSNS6O5Cek3Ffzu1Q 2014-07-06 07:34:13,097 INFO : org.elasticsearch.node - [graylog2-server] started 2014-07-06 07:34:13,172 INFO : org.elasticsearch.cluster.service - [graylog2-server] >detected_master [Silly Seal][neWyp2XsRFqzzcH$ 2014-07-06 07:34:18,102 ERROR: org.graylog2.Main -

>

ERROR: Could not successfully connect to ElasticSearch. Check that your cluster state is not RED >and that ElasticSearch is runnin$

2014-07-06 07:32:05,969 INFO : org.graylog2.Main - Graylog2 0.20.4 starting up. (JRE: Oracle >Corporation 1.7.0_55 on Linux 2.6.32$ 2014-07-06 07:33:17,041 INFO : org.graylog2.plugin.system.NodeId - Node ID: d375e0eb-07ff->494e-bf51-f17ea5be758a 2014-07-06 07:34:07,254 INFO : org.graylog2.buffers.ProcessBuffer - Initialized ProcessBuffer with >ring size <1024> and wait stra$ 2014-07-06 07:34:07,271 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with >ring size <1024> and wait strate$ 2014-07-06 07:34:07,796 INFO : org.elasticsearch.node - [graylog2-server] version[0.90.10], >pid[6811], build[0a5781f/2014-01-10T1$ 2014-07-06 07:34:07,796 INFO : org.elasticsearch.node - [graylog2-server] initializing ... 2014-07-06 07:34:07,801 INFO : org.elasticsearch.plugins - [graylog2-server] loaded [], sites [] 2014-07-06 07:34:10,031 INFO : org.elasticsearch.node - [graylog2-server] initialized 2014-07-06 07:34:10,031 INFO : org.elasticsearch.node - [graylog2-server] starting ... 2014-07-06 07:34:10,092 INFO : org.elasticsearch.transport - [graylog2-server] bound_address >{inet[/0.0.0.0:9350]}, publish_addre$ 2014-07-06 07:34:13,096 WARN : org.elasticsearch.discovery - [graylog2-server] waited for 3s and >no initial state was set by the $ 2014-07-06 07:34:13,097 INFO : org.elasticsearch.discovery - [graylog2-server] >graylog2/BezylqSNS6O5Cek3Ffzu1Q 2014-07-06 07:34:13,097 INFO : org.elasticsearch.node - [graylog2-server] started 2014-07-06 07:34:13,172 INFO : org.elasticsearch.cluster.service - [graylog2-server] >detected_master [Silly Seal][neWyp2XsRFqzzcH$ 2014-07-06 07:34:18,102 ERROR: org.graylog2.Main -

>

ERROR: Could not successfully connect to ElasticSearch. Check that your cluster state is not RED >and that ElasticSearch is runnin$

Elasticsearch, mongod and graylog2-web-interface are running fine, so I'm thinking that this may be down to communication between graylog2-server and elasticsearch

This in mind I took a look at /etc/graylog2.conf and noticed that the line

elasticsearch_cluster_name = graylog2

was previously commented out - I've removed this comment after double checking the cluster name in /etc/elasticsearch/elasticsearch.yml and I now get a different error (though the same ultimate result - graylog2-server fails to start)

2014-07-06 07:47:57,274 INFO : org.graylog2.Main - Graylog2 0.20.4 starting up. (JRE: Oracle >Corporation 1.7.0_55 on Linux 2.6.32$ 2014-07-06 07:47:57,382 INFO : org.graylog2.plugin.system.NodeId - Node ID: d375e0eb-07ff->494e-bf51-f17ea5be758a 2014-07-06 07:47:57,687 WARN : com.mongodb.tcp - Exception executing isMaster command on >/127.0.0.1:27017 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at com.mongodb.DBPort._open(DBPort.java:223) at com.mongodb.DBPort.go(DBPort.java:125) at com.mongodb.DBPort.go(DBPort.java:106) at com.mongodb.DBPort.findOne(DBPort.java:162) at com.mongodb.DBPort.runCommand(DBPort.java:170) at com.mongodb.DBTCPConnector.initDirectConnection(DBTCPConnector.java:533) at com.mongodb.DBTCPConnector.isMongosConnection(DBTCPConnector.java:334) at com.mongodb.Mongo.isMongosConnection(Mongo.java:618) at com.mongodb.DBCursor._check(DBCursor.java:364) at com.mongodb.DBCursor._hasNext(DBCursor.java:459) at com.mongodb.DBCursor.hasNext(DBCursor.java:484) at org.graylog2.database.Persisted.cursorToList(Persisted.java:106) at org.graylog2.database.Persisted.query(Persisted.java:78) at org.graylog2.dashboards.Dashboard.all(Dashboard.java:86) at org.graylog2.dashboards.DashboardRegistry.loadPersisted(DashboardRegistry.java:43) at org.graylog2.Core.initialize(Core.java:226) at org.graylog2.Main.main(Main.java:180) 2014-07-06 07:47:57,690 WARN : com.mongodb.tcp - Exception executing isMaster command on >/127.0.0.1:27017 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at com.mongodb.DBPort._open(DBPort.java:223) at com.mongodb.DBPort.go(DBPort.java:125) at com.mongodb.DBPort.go(DBPort.java:106) at com.mongodb.DBPort.findOne(DBPort.java:162) at com.mongodb.DBPort.runCommand(DBPort.java:170) at com.mongodb.DBTCPConnector.initDirectConnection(DBTCPConnector.java:533) at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:512) at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:236) 2014-07-06 07:47:57,274 INFO : org.graylog2.Main - Graylog2 0.20.4 starting up. (JRE: Oracle >Corporation 1.7.0_55 on Linux 2.6.32$ 2014-07-06 07:47:57,382 INFO : org.graylog2.plugin.system.NodeId - Node ID: d375e0eb-07ff->494e-bf51-f17ea5be758a 2014-07-06 07:47:57,687 WARN : com.mongodb.tcp - Exception executing isMaster command on >/127.0.0.1:27017 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at com.mongodb.DBPort._open(DBPort.java:223) at com.mongodb.DBPort.go(DBPort.java:125) at com.mongodb.DBPort.go(DBPort.java:106) at com.mongodb.DBPort.findOne(DBPort.java:162) at com.mongodb.DBPort.runCommand(DBPort.java:170) at com.mongodb.DBTCPConnector.initDirectConnection(DBTCPConnector.java:533) at com.mongodb.DBTCPConnector.isMongosConnection(DBTCPConnector.java:334) at com.mongodb.Mongo.isMongosConnection(Mongo.java:618) at com.mongodb.DBCursor._check(DBCursor.java:364) at com.mongodb.DBCursor._hasNext(DBCursor.java:459) at com.mongodb.DBCursor.hasNext(DBCursor.java:484) at org.graylog2.database.Persisted.cursorToList(Persisted.java:106) at org.graylog2.database.Persisted.query(Persisted.java:78) at org.graylog2.dashboards.Dashboard.all(Dashboard.java:86) at org.graylog2.dashboards.DashboardRegistry.loadPersisted(DashboardRegistry.java:43) at org.graylog2.Core.initialize(Core.java:226) at org.graylog2.Main.main(Main.java:180) 2014-07-06 07:47:57,690 WARN : com.mongodb.tcp - Exception executing isMaster command on >/127.0.0.1:27017 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at com.mongodb.DBPort._open(DBPort.java:223) at com.mongodb.DBPort.go(DBPort.java:125) at com.mongodb.DBPort.go(DBPort.java:106) at com.mongodb.DBPort.findOne(DBPort.java:162) at com.mongodb.DBPort.runCommand(DBPort.java:170) at com.mongodb.DBTCPConnector.initDirectConnection(DBTCPConnector.java:533) at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:512) at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:236) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216) at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:288) at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:273) at com.mongodb.DBCursor._check(DBCursor.java:368) at com.mongodb.DBCursor._hasNext(DBCursor.java:459) at com.mongodb.DBCursor.hasNext(DBCursor.java:484) at org.graylog2.database.Persisted.cursorToList(Persisted.java:106) at org.graylog2.database.Persisted.query(Persisted.java:78) at org.graylog2.dashboards.Dashboard.all(Dashboard.java:86) at org.graylog2.dashboards.DashboardRegistry.loadPersisted(DashboardRegistry.java:43) at org.graylog2.Core.initialize(Core.java:226) at org.graylog2.Main.main(Main.java:180) 2014-07-06 07:47:57,714 WARN : com.mongodb - emptying DBPortPool to /127.0.0.1:27017 b/c of >error java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at com.mongodb.DBPort._open(DBPort.java:223) at com.mongodb.DBPort.go(DBPort.java:125) at com.mongodb.DBPort.call(DBPort.java:92) at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:244) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216) at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:288) at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:273) at com.mongodb.DBCursor._check(DBCursor.java:368) at com.mongodb.DBCursor._hasNext(DBCursor.java:459) at com.mongodb.DBCursor.hasNext(DBCursor.java:484) at org.graylog2.database.Persisted.cursorToList(Persisted.java:106) at org.graylog2.database.Persisted.query(Persisted.java:78) at org.graylog2.dashboards.Dashboard.all(Dashboard.java:86) at org.graylog2.dashboards.DashboardRegistry.loadPersisted(DashboardRegistry.java:43) at org.graylog2.Core.initialize(Core.java:226) at org.graylog2.Main.main(Main.java:180) Exception in thread "main" com.mongodb.MongoException$Network: Read operation to server >/127.0.0.1:27017 failed on database grayl$ at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:253) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216) at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:288) at com.mongodb.DBApiLayer$MyCollection.find(DBApiLayer.java:273) at com.mongodb.DBCursor._check(DBCursor.java:368) at com.mongodb.DBCursor._hasNext(DBCursor.java:459) at com.mongodb.DBCursor.hasNext(DBCursor.java:484) at org.graylog2.database.Persisted.cursorToList(Persisted.java:106) at org.graylog2.database.Persisted.query(Persisted.java:78) at org.graylog2.dashboards.Dashboard.all(Dashboard.java:86) at org.graylog2.dashboards.DashboardRegistry.loadPersisted(DashboardRegistry.java:43) at org.graylog2.Core.initialize(Core.java:226) at org.graylog2.Main.main(Main.java:180) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at com.mongodb.DBPort._open(DBPort.java:223) at com.mongodb.DBPort.go(DBPort.java:125) at com.mongodb.DBPort.call(DBPort.java:92) at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:244) ... 12 more

You'll probably notice that I'm running 0.20.4 - I was seeing the exact same errors with 0.20.3, but thought it worth editing the script (literally just changing the version numbers pulled from the Graylog repo) and seeing if the issue was there in the latest version.

Working from the above errors I saw that you'd found issues with the service only listening on IPv6 interfaces, so I amended

$NOHUP java -jar ${GRAYLOG2_SERVER_JAR} -f ${GRAYLOG2_CONF} -p ${GRAYLOG2_PID} >> ${LOG_FILE} &

to read

$NOHUP java -Djava.net.preferIPv4Stack=true -jar ${GRAYLOG2_SERVER_JAR} -f >${GRAYLOG2_CONF} -p ${GRAYLOG2_PID} >> ${LOG_FILE} &

Restarted the server however still no dice.

2014-07-06 08:18:01,734 INFO : org.graylog2.Main - Graylog2 0.20.4 starting up. (JRE: Oracle >Corporation 1.7.0_55 on Linux 2.6.32$ 2014-07-06 08:18:01,773 INFO : org.graylog2.plugin.system.NodeId - Node ID: d375e0eb-07ff->494e-bf51-f17ea5be758a 2014-07-06 08:18:01,882 INFO : org.graylog2.buffers.ProcessBuffer - Initialized ProcessBuffer with >ring size <1024> and wait stra$ 2014-07-06 08:18:01,892 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with >ring size <1024> and wait strate$ 2014-07-06 08:18:02,238 INFO : org.elasticsearch.node - [graylog2-server] version[0.90.10], >pid[1934], build[0a5781f/2014-01-10T1$ 2014-07-06 08:18:02,238 INFO : org.elasticsearch.node - [graylog2-server] initializing ... 2014-07-06 08:18:02,242 INFO : org.elasticsearch.plugins - [graylog2-server] loaded [], sites [] 2014-07-06 08:18:03,539 INFO : org.elasticsearch.node - [graylog2-server] initialized 2014-07-06 08:18:03,539 INFO : org.elasticsearch.node - [graylog2-server] starting ... 2014-07-06 08:18:03,592 INFO : org.elasticsearch.transport - [graylog2-server] bound_address {inet[/0.0.0.0:9350]}, publish_addre$ 2014-07-06 08:18:06,596 WARN : org.elasticsearch.discovery - [graylog2-server] waited for 3s and >no initial state was set by the $ 2014-07-06 08:18:06,596 INFO : org.elasticsearch.discovery - [graylog2-server] >graylog2/t0e4EtU6Rgm0alBJQMWW0A 2014-07-06 08:18:06,597 INFO : org.elasticsearch.node - [graylog2-server] started 2014-07-06 08:18:06,651 INFO : org.elasticsearch.cluster.service - [graylog2-server] >detected_master [Comet][o1vDp6i1RCaZjdHyS5Jx$ 2014-07-06 08:18:11,602 ERROR: org.graylog2.Main -

>

ERROR: Could not successfully connect to ElasticSearch. Check that your cluster state is not RED >and that ElasticSearch is runnin$

Need help?

But we also got some specific help pages that might help you in this case:

Terminating. :(

>

As I say, this is purely with the CentOS script: Ubuntu script is working like a dream, but I need to be running on CentOS I'm afraid. Any ideas?

mrlesmithjr commented 10 years ago

init scipts for graylog2-server and elasticsearch changed. changed so that elasticsearch will start first