search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.41k stars 1.07k forks source link

Unable to upgrade to graylog 3.3 #9012

Closed RedMining closed 4 years ago

RedMining commented 4 years ago

Graylog fails to start upfer upgrade to version 3.3 to 3.2

Expected Behavior

Graylog starting to work and become available

Current Behavior

Fails with error log 

2020-09-22T11:25:34.551+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException
2020-09-22T11:25:34.551+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException
2020-09-22T11:25:34.551+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException
2020-09-22T11:25:34.551+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException
2020-09-22T11:25:34.551+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException
2020-09-22T11:25:34.552+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException
2020-09-22T11:25:34.552+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException
2020-09-22T11:25:34.552+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException
2020-09-22T11:25:34.552+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException
2020-09-22T11:25:34.552+02:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NullPointerException

Reading the messages log from the server I can find these messages:

Sep 22 11:25:34 hostname graylog-server: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:
Sep 22 11:25:34 hostname graylog-server: 1) Error injecting constructor, java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.buffers.ProcessBuffer.<init>(ProcessBuffer.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:59)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.buffers.ProcessBuffer
Sep 22 11:25:34 hostname graylog-server: for the 2nd parameter of org.graylog2.periodical.ThrottleStateUpdaterThread.<init>(ThrottleStateUpdaterThread.java:76)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.periodical.ThrottleStateUpdaterThread
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.plugin.periodical.Periodical annotated with @com.google.inject.internal.Element(setName=,uniqueId=378, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<org.graylog2.plugin.periodical.Periodical>
Sep 22 11:25:34 hostname graylog-server: for the 3rd parameter of org.graylog2.shared.initializers.PeriodicalsService.<init>(PeriodicalsService.java:45)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.initializers.PeriodicalsService.class(PeriodicalsService.java:36)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.initializers.PeriodicalsService
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.Service annotated with @com.google.inject.internal.Element(setName=,uniqueId=6, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<com.google.common.util.concurrent.Service>
Sep 22 11:25:34 hostname graylog-server: for field at org.graylog2.shared.bindings.providers.ServiceManagerProvider.services(ServiceManagerProvider.java:34)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.bindings.providers.ServiceManagerProvider
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:63)
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.ServiceManager
Sep 22 11:25:34 hostname graylog-server: Caused by: java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.buildFingerprint(StreamListFingerprint.java:55)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.<init>(StreamListFingerprint.java:37)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine.<init>(StreamRouterEngine.java:83)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine$$FastClassByGuice$$c6190f39.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836)
Sep 22 11:25:34 hostname graylog-server: at com.sun.proxy.$Proxy43.create(Unknown Source)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.getNewEngine(StreamRouter.java:136)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.access$000(StreamRouter.java:102)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter.<init>(StreamRouter.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$$FastClassByGuice$$f9455617.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:
Sep 22 11:25:34 hostname graylog-server: 1) Error injecting constructor, java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.buffers.ProcessBuffer.<init>(ProcessBuffer.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:59)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.buffers.ProcessBuffer
Sep 22 11:25:34 hostname graylog-server: for the 2nd parameter of org.graylog2.periodical.ThrottleStateUpdaterThread.<init>(ThrottleStateUpdaterThread.java:76)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.periodical.ThrottleStateUpdaterThread
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.plugin.periodical.Periodical annotated with @com.google.inject.internal.Element(setName=,uniqueId=378, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<org.graylog2.plugin.periodical.Periodical>
Sep 22 11:25:34 hostname graylog-server: for the 3rd parameter of org.graylog2.shared.initializers.PeriodicalsService.<init>(PeriodicalsService.java:45)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.initializers.PeriodicalsService.class(PeriodicalsService.java:36)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.initializers.PeriodicalsService
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.Service annotated with @com.google.inject.internal.Element(setName=,uniqueId=6, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<com.google.common.util.concurrent.Service>
Sep 22 11:25:34 hostname graylog-server: for field at org.graylog2.shared.bindings.providers.ServiceManagerProvider.services(ServiceManagerProvider.java:34)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.bindings.providers.ServiceManagerProvider
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:63)
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.ServiceManager
Sep 22 11:25:34 hostname graylog-server: Caused by: java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.buildFingerprint(StreamListFingerprint.java:55)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.<init>(StreamListFingerprint.java:37)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine.<init>(StreamRouterEngine.java:83)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine$$FastClassByGuice$$c6190f39.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836)
Sep 22 11:25:34 hostname graylog-server: at com.sun.proxy.$Proxy43.create(Unknown Source)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.getNewEngine(StreamRouter.java:136)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.access$000(StreamRouter.java:102)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter.<init>(StreamRouter.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$$FastClassByGuice$$f9455617.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:
Sep 22 11:25:34 hostname graylog-server: 1) Error injecting constructor, java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.buffers.ProcessBuffer.<init>(ProcessBuffer.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:59)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.buffers.ProcessBuffer
Sep 22 11:25:34 hostname graylog-server: for the 2nd parameter of org.graylog2.periodical.ThrottleStateUpdaterThread.<init>(ThrottleStateUpdaterThread.java:76)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.periodical.ThrottleStateUpdaterThread
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.plugin.periodical.Periodical annotated with @com.google.inject.internal.Element(setName=,uniqueId=378, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<org.graylog2.plugin.periodical.Periodical>
Sep 22 11:25:34 hostname graylog-server: for the 3rd parameter of org.graylog2.shared.initializers.PeriodicalsService.<init>(PeriodicalsService.java:45)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.initializers.PeriodicalsService.class(PeriodicalsService.java:36)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.initializers.PeriodicalsService
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.Service annotated with @com.google.inject.internal.Element(setName=,uniqueId=6, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<com.google.common.util.concurrent.Service>
Sep 22 11:25:34 hostname graylog-server: for field at org.graylog2.shared.bindings.providers.ServiceManagerProvider.services(ServiceManagerProvider.java:34)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.bindings.providers.ServiceManagerProvider
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:63)
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.ServiceManager
Sep 22 11:25:34 hostname graylog-server: Caused by: java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.buildFingerprint(StreamListFingerprint.java:55)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.<init>(StreamListFingerprint.java:37)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine.<init>(StreamRouterEngine.java:83)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine$$FastClassByGuice$$c6190f39.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836)
Sep 22 11:25:34 hostname graylog-server: at com.sun.proxy.$Proxy43.create(Unknown Source)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.getNewEngine(StreamRouter.java:136)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.access$000(StreamRouter.java:102)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter.<init>(StreamRouter.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$$FastClassByGuice$$f9455617.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:
Sep 22 11:25:34 hostname graylog-server: 1) Error injecting constructor, java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.buffers.ProcessBuffer.<init>(ProcessBuffer.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:59)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.buffers.ProcessBuffer
Sep 22 11:25:34 hostname graylog-server: for the 2nd parameter of org.graylog2.periodical.ThrottleStateUpdaterThread.<init>(ThrottleStateUpdaterThread.java:76)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.periodical.ThrottleStateUpdaterThread
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.plugin.periodical.Periodical annotated with @com.google.inject.internal.Element(setName=,uniqueId=378, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<org.graylog2.plugin.periodical.Periodical>
Sep 22 11:25:34 hostname graylog-server: for the 3rd parameter of org.graylog2.shared.initializers.PeriodicalsService.<init>(PeriodicalsService.java:45)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.initializers.PeriodicalsService.class(PeriodicalsService.java:36)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.initializers.PeriodicalsService
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.Service annotated with @com.google.inject.internal.Element(setName=,uniqueId=6, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<com.google.common.util.concurrent.Service>
Sep 22 11:25:34 hostname graylog-server: for field at org.graylog2.shared.bindings.providers.ServiceManagerProvider.services(ServiceManagerProvider.java:34)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.bindings.providers.ServiceManagerProvider
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:63)
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.ServiceManager
Sep 22 11:25:34 hostname graylog-server: Caused by: java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.buildFingerprint(StreamListFingerprint.java:55)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.<init>(StreamListFingerprint.java:37)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine.<init>(StreamRouterEngine.java:83)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine$$FastClassByGuice$$c6190f39.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836)
Sep 22 11:25:34 hostname graylog-server: at com.sun.proxy.$Proxy43.create(Unknown Source)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.getNewEngine(StreamRouter.java:136)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.access$000(StreamRouter.java:102)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter.<init>(StreamRouter.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$$FastClassByGuice$$f9455617.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:
Sep 22 11:25:34 hostname graylog-server: 1) Error injecting constructor, java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.buffers.ProcessBuffer.<init>(ProcessBuffer.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:59)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.buffers.ProcessBuffer
Sep 22 11:25:34 hostname graylog-server: for the 2nd parameter of org.graylog2.periodical.ThrottleStateUpdaterThread.<init>(ThrottleStateUpdaterThread.java:76)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.periodical.ThrottleStateUpdaterThread
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.plugin.periodical.Periodical annotated with @com.google.inject.internal.Element(setName=,uniqueId=378, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<org.graylog2.plugin.periodical.Periodical>
Sep 22 11:25:34 hostname graylog-server: for the 3rd parameter of org.graylog2.shared.initializers.PeriodicalsService.<init>(PeriodicalsService.java:45)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.initializers.PeriodicalsService.class(PeriodicalsService.java:36)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.initializers.PeriodicalsService
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.Service annotated with @com.google.inject.internal.Element(setName=,uniqueId=6, type=MULTIBINDER, keyType=)
Sep 22 11:25:34 hostname graylog-server: while locating java.util.Set<com.google.common.util.concurrent.Service>
Sep 22 11:25:34 hostname graylog-server: for field at org.graylog2.shared.bindings.providers.ServiceManagerProvider.services(ServiceManagerProvider.java:34)
Sep 22 11:25:34 hostname graylog-server: while locating org.graylog2.shared.bindings.providers.ServiceManagerProvider
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.shared.bindings.GenericBindings.configure(GenericBindings.java:63)
Sep 22 11:25:34 hostname graylog-server: while locating com.google.common.util.concurrent.ServiceManager
Sep 22 11:25:34 hostname graylog-server: Caused by: java.lang.NullPointerException
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.buildFingerprint(StreamListFingerprint.java:55)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamListFingerprint.<init>(StreamListFingerprint.java:37)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine.<init>(StreamRouterEngine.java:83)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouterEngine$$FastClassByGuice$$c6190f39.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836)
Sep 22 11:25:34 hostname graylog-server: at com.sun.proxy.$Proxy43.create(Unknown Source)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.getNewEngine(StreamRouter.java:136)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$StreamRouterEngineUpdater.access$000(StreamRouter.java:102)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter.<init>(StreamRouter.java:62)
Sep 22 11:25:34 hostname graylog-server: at org.graylog2.streams.StreamRouter$$FastClassByGuice$$f9455617.newInstance(<generated>)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
Sep 22 11:25:34 hostname graylog-server: at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)

Possible Solution

Unkwown

Steps to Reproduce (for bugs)

I've create a brand new server and installed a 3.3 graylog server and connected to the old cluster version 3.2 but the error occurred again. I've tried before to upgrade the cluster, but had to rollback because unable to start the graylog instance.

If necessary, I can upload the mongo and ElasticSearch conf

Context

I can't upgrade graylog to the latest availble version

Your Environment

dennisoelkers commented 4 years ago

Do you had any streams and/or outputs configured before the upgrade?

RedMining commented 4 years ago

Yeah, it's a production cluster, so we have many input and one output and obviously some streams

jalogisch commented 4 years ago

@RedMining what is the content of your plugin folder?

You can't run Graylog cluster with mixed versions using the same database (MongoDB), but I guess that some old plugins are causing the problems.

RedMining commented 4 years ago

@jalogisch I've tried emptying the folder and then upgrade the cluster, but the result didn't change Here the content of our plugin folder:


graylog-plugin-alert-wizard-1.1.2.jar
graylog-plugin-aws-3.2.6.jar
graylog-plugin-collector-3.2.6.jar
graylog-plugin-logging-alert-1.0.1.jar
graylog-plugin-rundeck-1.2.1.jar
graylog-plugin-slack-3.0.1.jar
graylog-plugin-threatintel-3.2.6.jar
original-graylog-delimited-file-output-plugin-0.1.0.jar
original-graylog-plugin-file-output-1.0.1-SNAPSHOT.jar
telegram-alert-2.1.0.jar```
jalogisch commented 4 years ago

he @RedMining

please remove all plugins and start Graylog - if it is working correctly please ingest plugins in a compatible version one after another.

I can spot minimal 3 plugins with the wrong version for your desired Graylog version.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

dennisoelkers commented 4 years ago

Besides the plugin issue that @jalogisch brought up, we also might have another issue. @RedMining: is there a chance to get your streams collection from your mongodb? If you do not want to post it publicly, you could also send it to dennis@graylog.com.

RedMining commented 4 years ago

@jalogisch I've already tried to upgrade the cluster without any plugin, but I got the same error. I've tried just now it and had to rollback. @dennisoelkers I don't know how to extract that info from mongo, if you could help me I would be glad to give you the information

RedMining commented 4 years ago

@dennisoelkers If it could help you, I've downloaded the OVA and started on my pc and imported a content packs with all the inputs, dashboard and streams from our production cluster, import and install went without problems.

dennisoelkers commented 4 years ago

@RedMining: You can use mongoexport to export the streams collection. An example command line would be:

mongoexport --uri="mongodb://mongo.example.com:27017/graylog"  --collection=streams  --out=streams.json

Where your mongodb would be running on port 27017 of mongo.example.com, the db name would be graylog and the resulting file that I would be interested in is streams.json.

RedMining commented 4 years ago

Hi dennisoelkers, In attachment you can find the export. streams.json.zip

dennisoelkers commented 4 years ago

Thanks, @RedMining! Can you do the same for the outputs collection? I am suspecting that some of the outputs which are referenced from your streams are missing, for unknown reasons. Did you reuse the same MongoDB when upgrading from 3.2 to 3.3? Did you copy it over? If yes, how?

RedMining commented 4 years ago

outputs.json.zip

Hi @dennisoelkers In attach you can find the extraction for the output. When tried to upgrade I've used the same mongo, changing anything. There are some action as prerequisites that I missed? Maybe I can try to update removing the output, and if successful try to configure it later

dennisoelkers commented 4 years ago

Thanks a lot. In your collection seems to be only a single output, while four are referenced in total by your streams. As this should not block server startup, I have created a PR (#9116) to ignore this, but it will take a while to be merged and released.

In the meantime, you can remove the references in your streams collection to make startup work again. For reference, this is the output that I get with the PR:

2020-10-08 14:49:41,342 WARN : org.graylog2.streams.StreamServiceImpl - Stream "auditd.log" <5ce7d07700f4663aab279967> references missing output <5d9217c000f46655d0458931> - ignoring output.
2020-10-08 14:49:41,342 WARN : org.graylog2.streams.StreamServiceImpl - Stream "SSHD.log" <5ce7d08700f4663aab27997a> references missing output <5d92013f00f46667bf15ecbd> - ignoring output.
2020-10-08 14:49:41,342 WARN : org.graylog2.streams.StreamServiceImpl - Stream "SSHD.log" <5ce7d08700f4663aab27997a> references missing output <5d921ce500f46648c7a594ce> - ignoring output.

So what you would need to do is edit your streams collection manually and remove the references from the SSHD/audit streams.

The remaining question is: How did you manage to delete outputs without the references getting pruned from the streams? I have no idea, because removing an output should remove all of its references too. Do you have any idea?

RedMining commented 4 years ago

@dennisoelkers you're great man! That was the error. I've removed from mongo the reference to the missing outputs and alerts and updated without problems. I don't know how it happend, I've never touched mongo before, only for upgrading to version 4.0, but this problem was already present. By my side, the issue was resolved
Thanks again for your support!

dennisoelkers commented 4 years ago

You are welcome! I am glad I was able to help you.

One note on the side: If you are working with Graylog in a commercial environment, please consider our Technical Support Offerings. We are offering world class support to help you run Graylog in a scalable and professional fashion. Obviously, the money spent for support is directly funding the development of Graylog.

I am closing this issue. Feel free to reopen it if you have anything to add.

stefangweichinger commented 2 years ago

I have a similar issue and see 0 outputs in my mongoexport (!)