search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.33k stars 1.05k forks source link

Improperly escaped square brackets in a pipeline rule's grok function do not throw a validation error #14818

Closed damianharouff closed 1 year ago

damianharouff commented 1 year ago

The following pipeline rule will not produce an in-context validation error, nor can the pipeline rule be saved:

rule "customer haproxy log parse" when from_input("63fe0a0c9d42820718a38fea") then let raw_log=to_string($message.message); let my_grok=grok( pattern:"(<...>%{SYSLOGTIMESTAMP:timestamp} %{NOTSPACE:source} haproxy\[%{NUMBER}\]\: %{IP:source_ip}:%{NUMBER:source_port} %{NOTSPACE} %{WORD}. %{WORD}/%{WORD} %{NOTSPACE} %{NUMBER:resp_code} %{GREEDYDATA}{%{DATA:user_agent}} \"%{WORD:method} %{NOTSPACE:uri} %{WORD:protocol}/%{BASE10NUM:http_ver}\")", value: raw_log, only_named_captures: true ); end

image

While we can note that the issue is due to lack of double backslashes on the square brackets, the end user is left wondering why nothing is functional, since user is neither getting an in-context error, nor can save the rule.

Every time the Save button is clicked, the following stack trace is generated in server.log:

2023-02-28T12:54:27.516-07:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource java.lang.NullPointerException: null at java.util.stream.MatchOps$1MatchSink.accept(Unknown Source) ~[?:?] at java.util.HashMap$ValueSpliterator.tryAdvance(Unknown Source) ~[?:?] at java.util.stream.ReferencePipeline.forEachWithCancel(Unknown Source) ~[?:?] at java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown Source) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?] at java.util.stream.MatchOps$MatchOp.evaluateSequential(Unknown Source) ~[?:?] at java.util.stream.MatchOps$MatchOp.evaluateSequential(Unknown Source) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?] at java.util.stream.ReferencePipeline.allMatch(Unknown Source) ~[?:?] at org.graylog.plugins.pipelineprocessor.ast.expressions.MapLiteralExpression.isConstant(MapLiteralExpression.java:41) ~[graylog.jar:?] at org.graylog.plugins.pipelineprocessor.ast.functions.FunctionArgs.lambda$getConstantArgs$0(FunctionArgs.java:57) ~[graylog.jar:?] at java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) ~[?:?] at java.util.HashMap$EntrySpliterator.forEachRemaining(Unknown Source) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?] at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?] at org.graylog.plugins.pipelineprocessor.ast.functions.FunctionArgs.getConstantArgs(FunctionArgs.java:59) ~[graylog.jar:?] at org.graylog.plugins.pipelineprocessor.ast.functions.Function.preprocessArgs(Function.java:54) ~[graylog.jar:?] at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.<init>(FunctionExpression.java:44) ~[graylog.jar:?] at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser$RuleAstBuilder.exitFunctionCall(PipelineRuleParser.java:394) ~[graylog.jar:?] at org.graylog.plugins.pipelineprocessor.parser.RuleLangParser$FunctionCallContext.exitRule(RuleLangParser.java:1478) ~[graylog.jar:?] at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:47) ~[graylog.jar:?] at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:30) ~[graylog.jar:?] at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[graylog.jar:?] at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[graylog.jar:?] at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[graylog.jar:?] at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[graylog.jar:?] at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:156) ~[graylog.jar:?] at org.graylog.plugins.pipelineprocessor.rest.RuleResource.parse(RuleResource.java:163) ~[graylog.jar:?] at jdk.internal.reflect.GeneratedMethodAccessor1793.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?] at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[graylog.jar:?] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [graylog.jar:?] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?] at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?]

However, this was brought to our attention via HS-1471224951, which is a Cloud customer, and Cloud customers cannot see their own server.log.

Tested in: Graylog 5.0.3+a82acb2 (Eclipse Adoptium 17.0.6 on Linux 5.10.0-21-amd64) Graylog Cloud 5.0.4 (348) Graylog Cloud 5.0.4 (361)

moesterheld commented 1 year ago

@damianharouff the issue here was triggered by the single backslash, but actually caused by the thus parsed string containing an unnamed grok group haproxy\[%{NUMBER} (oh yes, and the trailing \"was also needed for that to happen)

The pipeline parser thought the grok pattern was a map with a key but no value, causing the exception. It turned out that this is a more general problem as this error is also thrown when entering maps in the rule editor.

16175 takes care of the issue.