search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.43k stars 1.07k forks source link

Error Starting Graylog after upgrade to 4.0.1 #9781

Closed hilmar123 closed 3 years ago

hilmar123 commented 3 years ago

The graylog service is stuck in a launch -> running -> crash -> launch.... loop, the service status is as follows:

● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2020-12-11 12:23:37 CET; 2s ago
     Docs: http://docs.graylog.org/
  Process: 6210 ExecStart=/usr/share/graylog-server/bin/graylog-server (code=exited, status=1/FAILURE)
 Main PID: 6210 (code=exited, status=1/FAILURE)

Here is the last lines from the graylog-server log file:

2020-12-11T13:05:29.004+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2020-12-11T13:05:29.207+01:00 INFO  [Version] HV000001: Hibernate Validator null
2020-12-11T13:05:41.468+01:00 WARN  [DefaultTLSProtocolProvider] JRE doesn't support all default TLS protocols. Changing <[TLSv1.2, TLSv1.3]> to <[TLSv1.2]>
2020-12-11T13:05:41.690+01:00 INFO  [CmdLineTool] Loaded plugin: Aggregation Count Alert Condition 1.1.0 [com.airbus-cyber-security.graylog.AggregationCountPlugin]
2020-12-11T13:05:41.691+01:00 INFO  [CmdLineTool] Loaded plugin: Correlation Count Alert Condition 1.0.1 [com.airbus-cyber-security.graylog.CorrelationCountPlugin]
2020-12-11T13:05:41.691+01:00 INFO  [CmdLineTool] Loaded plugin: Logging Alert Notification 1.0.0 [com.airbus-cyber-security.graylog.LoggingAlertPlugin]
2020-12-11T13:05:41.692+01:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.0.1 [org.graylog.aws.AWSPlugin]
2020-12-11T13:05:41.693+01:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integrations 4.0.1 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2020-12-11T13:05:41.693+01:00 INFO  [CmdLineTool] Loaded plugin: Integrations 4.0.1 [org.graylog.integrations.IntegrationsPlugin]
2020-12-11T13:05:41.694+01:00 INFO  [CmdLineTool] Loaded plugin: Collector 4.0.1 [org.graylog.plugins.collector.CollectorPlugin]
2020-12-11T13:05:41.694+01:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise 4.0.1 [org.graylog.plugins.enterprise.EnterprisePlugin]
2020-12-11T13:05:41.695+01:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.0.0 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2020-12-11T13:05:41.695+01:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.0.1+6a0cc0b [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2020-12-11T13:05:41.695+01:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.0.1+6a0cc0b [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2020-12-11T13:05:41.872+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2020-12-11T13:05:42.078+01:00 INFO  [Version] HV000001: Hibernate Validator null

This repeats again and again, I cant see any special error message in this log. When I try to run the graylog service directly from /usr/share/graylog/server/bin I get the following:

graylog@graylog:/usr/share/graylog-server/bin$ ./graylog-server 
Exception in thread "main" java.lang.NoClassDefFoundError: org/graylog2/indexer/results/TermsResult
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.getDeclaredMethods(Class.java:1975)
        at com.google.inject.spi.InjectionPoint.getDeclaredMethods(InjectionPoint.java:766)
        at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:683)
        at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:378)
        at com.google.inject.assistedinject.FactoryProvider2.getDependencies(FactoryProvider2.java:615)
        at com.google.inject.assistedinject.FactoryProvider2.<init>(FactoryProvider2.java:325)
        at com.google.inject.assistedinject.FactoryModuleBuilder$1.configure(FactoryModuleBuilder.java:316)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:61)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:344)
        at com.google.inject.AbstractModule.install(AbstractModule.java:103)
        at org.graylog2.plugin.inject.Graylog2Module.installAlertConditionWithCustomName(Graylog2Module.java:388)
        at org.graylog2.plugin.PluginModule.addAlertCondition(PluginModule.java:204)
        at com.airbus_cyber_security.graylog.CorrelationCountModule.configure(CorrelationCountModule.java:23)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:61)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:344)
        at org.graylog2.shared.bindings.PluginBindings.configure(PluginBindings.java:51)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:61)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:344)
        at com.google.inject.spi.Elements.getElements(Elements.java:103)
        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:137)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103)
        at com.google.inject.Guice.createInjector(Guice.java:87)
        at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
        at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:381)
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:196)
        at org.graylog2.bootstrap.Main.main(Main.java:50)
Caused by: java.lang.ClassNotFoundException: org.graylog2.indexer.results.TermsResult
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:817)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 28 more

The line "Caused by: java.lang.ClassNotFoundException: org.graylog2.indexer.results.TermsResult" is the only message I have found that looks like a error, but I dont know what would cause that. As a side note, when I first upgraded from my previous 3.8 version to 4.0 it worked fine the first day, but when I checked back a couple of days later the service was down. Hope you can help.

Your Environment

hulkk commented 3 years ago

Did you check this #9752

hilmar123 commented 3 years ago

Sorry, I forgot to include that in my original post, here is the plugins in my plugin folder: graylog-plugin-aggregation-count-1.1.0.jar graylog-plugin-enterprise-integrations-4.0.1.jar graylog-storage-elasticsearch6-4.0.1.jar graylog-plugin-aws-4.0.1.jar graylog-plugin-integrations-4.0.1.jar graylog-storage-elasticsearch7-4.0.1.jar graylog-plugin-collector-4.0.1.jar graylog-plugin-logging-alert-1.0.0.jar LICENSE-ENTERPRISE graylog-plugin-correlation-count-1.0.1.jar graylog-plugin-threatintel-3.0.0.jar.old graylog-plugin-enterprise-4.0.1.jar graylog-plugin-threatintel-4.0.1.jar

There are the three files aggregation-count, correlation-count and logging-alert that doesn't have 4.0.1 version, I have tried to move them out of the plugin folder but the graylog-service still wont start.

EDIT: I double tried this now, just to make sure. Moved out all the -1 and -3 plugins, then I got the service to start. Now I cant log in though. Probably screwed up something else while trying to fix this. Thanks for the help

hulkk commented 3 years ago

Did you read the upgrade notes? LDAP is disabled and configurations has to be reviewed and enabled manually. You have to log in using admin account.

hilmar123 commented 3 years ago

Yah, I saw that now. Logged in and enabled the new LDAP system. Thanks a lot for the help.