search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.37k stars 1.06k forks source link

NPEs in new PipelineProcessor - Server does not start #3153

Closed lennartkoopmann closed 7 years ago

lennartkoopmann commented 7 years ago

I'm trying out 2.2.0-beta.1 on Debian but cannot start the server. It hangs with these stacktraces:

2016-12-05 11:47:26,859 DEBUG: org.graylog2.bootstrap.CmdLineTool - Stacktrace:
java.lang.IllegalStateException: null
    at org.graylog.plugins.pipelineprocessor.ast.Rule.registerMetrics(Rule.java:96) ~[?:?]
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.lambda$null$3(ConfigurationStateUpdater.java:182) ~[?:?]
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_111]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_111]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_111]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_111]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_111]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_111]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_111]
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.lambda$resolvePipeline$4(ConfigurationStateUpdater.java:185) ~[?:?]
    at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_111]
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.resolvePipeline(ConfigurationStateUpdater.java:170) ~[?:?]
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.lambda$reload$1(ConfigurationStateUpdater.java:139) ~[?:?]
    at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_111]
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.reload(ConfigurationStateUpdater.java:131) ~[?:?]
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.reloadAndSave(ConfigurationStateUpdater.java:108) ~[?:?]
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.<init>(ConfigurationStateUpdater.java:88) ~[?:?]
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater$$FastClassByGuice$$13de1e44.newInstance(<generated>) ~[?:?]
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[graylog.jar:?]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[graylog.jar:?]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[graylog.jar:?]
    at com.google.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:375) ~[graylog.jar:?]
    at com.google.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:258) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) ~[graylog.jar:?]
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61) ~[graylog.jar:?]
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[graylog.jar:?]
    at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:776) ~[graylog.jar:?]
    at com.sun.proxy.$Proxy89.create(Unknown Source) ~[?:?]
    at org.graylog2.shared.buffers.ProcessBuffer.<init>(ProcessBuffer.java:92) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.ProcessBuffer$$FastClassByGuice$$ef94431e.newInstance(<generated>) ~[graylog.jar:?]
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[graylog.jar:?]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[graylog.jar:?]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[graylog.jar:?]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[graylog.jar:?]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[graylog.jar:?]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[graylog.jar:?]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[graylog.jar:?]
    at com.google.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:375) ~[graylog.jar:?]
    at com.google.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:258) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) ~[graylog.jar:?]
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61) ~[graylog.jar:?]
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[graylog.jar:?]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[graylog.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[graylog.jar:?]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[graylog.jar:?]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[graylog.jar:?]
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205) ~[graylog.jar:?]
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199) ~[graylog.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[graylog.jar:?]
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199) ~[graylog.jar:?]
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180) ~[graylog.jar:?]
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) ~[graylog.jar:?]
    at com.google.inject.Guice.createInjector(Guice.java:99) ~[graylog.jar:?]
    at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34) ~[graylog.jar:?]
    at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379) [graylog.jar:?]
    at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:193) [graylog.jar:?]
    at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error injecting constructor, java.lang.IllegalStateException
  at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.<init>(ConfigurationStateUpdater.java:71)
  at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.class(ConfigurationStateUpdater.java:44)
  while locating org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater
Caused by: java.lang.IllegalStateException
    at org.graylog.plugins.pipelineprocessor.ast.Rule.registerMetrics(Rule.java:96)
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.lambda$null$3(ConfigurationStateUpdater.java:182)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.lambda$resolvePipeline$4(ConfigurationStateUpdater.java:185)
    at java.lang.Iterable.forEach(Iterable.java:75)
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.resolvePipeline(ConfigurationStateUpdater.java:170)
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.lambda$reload$1(ConfigurationStateUpdater.java:139)
    at java.lang.Iterable.forEach(Iterable.java:75)
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.reload(ConfigurationStateUpdater.java:131)
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.reloadAndSave(ConfigurationStateUpdater.java:108)
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater.<init>(ConfigurationStateUpdater.java:88)
    at org.graylog.plugins.pipelineprocessor.processors.ConfigurationStateUpdater$$FastClassByGuice$$13de1e44.newInstance(<generated>)
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
    at com.google.inject.Guice.createInjector(Guice.java:99)
    at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
    at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379)
    at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:193)
    at org.graylog2.bootstrap.Main.main(Main.java:44)

The server starts up successfully after deleting all pipeline_processor_* collections in MongoDB so I suspect the errors are triggered by one of my rules or pipeline configurations.

I have uploaded a mongodump of the collections for easier reproduction: graylog.zip (GitHub only allows .zip, sorry)

lennartkoopmann commented 7 years ago

This happens when running with or without JDK.

edmundoa commented 7 years ago

Probably solved in here https://github.com/Graylog2/graylog-plugin-pipeline-processor/pull/141

edmundoa commented 7 years ago

@lennartkoopmann could you take a quick look with the current master of the pipeline processor plugin? Let me know if you can't quickly test the plugin, and I will take a look at the mongodb dump you provided. Thank you!

lennartkoopmann commented 7 years ago

Sorry, don't have plugin builds setup currently. :/ Could you test it? Thanks! :)

edmundoa commented 7 years ago

Sure, will take a look 👍

edmundoa commented 7 years ago

Closing the issue, as I can confirm that this was happening with the pipeline processor plugin 2.1 code, but is not happening on master any more.

lennartkoopmann commented 7 years ago

Thank you! :)