search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.37k stars 1.06k forks source link

Graylog service failing after upgrade to 2.3.1 #4112

Closed ElZeeBee closed 7 years ago

ElZeeBee commented 7 years ago

After updating to version 2.3.1, the Graylog service is failing to start (or it runs very briefly before failing). journalctl suggests this may be related to Collector Sidecar.

Also posted to Graylog2/collector-sidecar#188

Expected Behavior

Graylog service should start and run as normal

Current Behavior

Graylog service fails to start (or runs briefly before failing). journalctrl reports:

Aug 25 13:18:45 centos7vm.localdomain systemd[1]: Started Graylog server.
Aug 25 13:18:45 centos7vm.localdomain systemd[1]: Starting Graylog server...
Aug 25 13:18:47 centos7vm.localdomain systemd[1]: graylog-server.service: main process exited, code=exited, status=1/FAILURE
Aug 25 13:18:47 centos7vm.localdomain systemd[1]: Unit graylog-server.service entered failed state.
Aug 25 13:18:47 centos7vm.localdomain systemd[1]: graylog-server.service failed.
Aug 25 13:18:49 centos7vm.localdomain graylog-collector-sidecar[6158]: time="2017-08-25T13:18:49-04:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://127.0.0.1:9000/api/plugins/org.graylog.plugins.collector/5d311f6a-c554-43f5-b0eb-817a5b5a92bc?tags=%5B%22linux%22%2C%22apache%22%2C%22itos%22%5D: dial tcp 127.0.0.1:9000: getsockopt: connection refused"
Aug 25 13:18:50 centos7vm.localdomain graylog-collector-sidecar[6158]: time="2017-08-25T13:18:50-04:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://127.0.0.1:9000/api/plugins/org.graylog.plugins.collector/collectors/5d311f6a-c554-43f5-b0eb-817a5b5a92bc: dial tcp 127.0.0.1:9000: getsockopt: connection refused"

Steps to Reproduce (for bugs)

  1. Update Graylog via yum
  2. Try to access web console
  3. Run systemtl status graylog-server.service
  4. Observe Graylog service failure

Context

Your Environment

Collector Sidecar 0.1.4 Filebeat 5.5.1

joschi commented 7 years ago

@ElZeeBee What's in the logs of your Graylog node? http://docs.graylog.org/en/2.3/pages/configuration/file_location.html

ElZeeBee commented 7 years ago

According to the logs, the only issue is that Parameter password_secret should not be blank. When I first upgraded from 2.2.x, the new config file was created with the new parameters as described in the upgrade guide in the documentation, but Graylog still seemed to work; I was able to bring up the web console and log in with the old config settings. Because of this, I didn't think to check the new config file.

I have added in password_secret and root_password_sha2 and it seems to be working now. Interesting note: I checked the service before updating the new config file, and it seemed to be running.

Logs for reference:


2017-08-25T14:04:41.520-04:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.3.1 [org.graylog.plugins.collector.CollectorPlugin]
2017-08-25T14:04:41.520-04:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.3.1 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-08-25T14:04:41.521-04:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.3.1 [org.graylog.plugins.map.MapWidgetPlugin]
2017-08-25T14:04:41.528-04:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.3.1 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-08-25T14:04:41.528-04:00 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.3.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-08-25T14:04:41.541-04:00 ERROR [CmdLineTool] Invalid configuration
com.github.joschi.jadconfig.ValidationException: Parameter password_secret should not be blank
at com.github.joschi.jadconfig.validators.StringNotBlankValidator.validate(StringNotBlankValidator.java:25) ~[graylog.jar:?]
at com.github.joschi.jadconfig.validators.StringNotBlankValidator.validate(StringNotBlankValidator.java:11) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.validateParameter(JadConfig.java:215) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:148) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:99) ~[graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:351) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.readConfiguration(CmdLineTool.java:344) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:177) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]