Unable to retrieve expression for variable file, this is a bug / Unhandled exception in REST resource

kronenpj commented 7 years ago

While attempting to add a pipeline rule, the UI emits the red error banner saying: Saving rule "" failed with status: cannot POST http://10.0.0.2:9000/api/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule (500)

Attempted rule addition:

rule "SELinux AVC Extraction"
when
  has_field("file") && file == "/var/log/audit/audit.log"
then
  grok("avc_extractor", $message)
end

The 'avc_extractor' grok was already loaded into Graylog.

Expected Behavior

Assuming syntax of pipeline rule is correct, accept the rule and make available via UI.

Current Behavior

While attempting to add a pipeline rule, the UI emits the red error banner saying: Saving rule "" failed with status: cannot POST http://10.0.0.2:9000/api/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule (500)

Graylog log file contains:

2017-09-28T08:14:20.556-04:00 ERROR [PipelineRuleParser] Unable to retrieve expression for variable file, this is a bug
2017-09-28T08:14:20.557-04:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.NullPointerException: null
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser$RuleTypeChecker.exitMessageRef(PipelineRuleParser.java:817) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.RuleLangParser$MessageRefContext.exitRule(RuleLangParser.java:713) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:172) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:135) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.rest.RuleResource.createFromParser(RuleResource.java:93) ~[?:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

Possible Solution

Sorry, can't help here.

Steps to Reproduce (for bugs)

Add grok from https://github.com/madchap/graylog2-selinux.git as 'avc_extractor'
Attempt to add above pipeline rule
Be disappointed that it triggers a bug.

Context

The grok is intended to parse the SELinux AVC messages collected from /var/log/audit/audit.log into usable fields. The pipeline rule would invoke the grok on appropriate messages.

Your Environment

Graylog Version: 2.4.0-beta3 (also 2.3.1)
Elasticsearch Version: 5.5.2-1.el7
MongoDB Version: 3.2.16-1.el7
Operating System: CentOS 7-1708
Browser version: Chrome 61.0.3163.100, Firefox 55.0.3

Also tried adding the rule via API Browser with the same result.

joschi commented 7 years ago

@kronenpj Your rule is syntactically incorrect.

Try this one:

rule "SELinux AVC Extraction"
when
  has_field("file") && to_string($message.file) == "/var/log/audit/audit.log"
then
  let matches = grok("%{avc_extractor}", $message.message);
  set_fields(matches);
end

This being said, a syntactically incorrect rule should never trigger a NullPointerException but a emit parse error instead.

Graylog Version: 2.4.0-beta3 (also 2.3.1)

There is no Graylog 2.4.0-beta3 (yet).

kronenpj commented 7 years ago

Quite right, alpha-3...

I hadn't thought to try the let/set_fields combination. I'm sure that'll work. Thanks!

kronenpj commented 7 years ago

For posterity, joschi's suggestion was very slightly incorrect. This syntax was accepted by the parser:

rule "SELinux AVC Extraction"
when
  has_field("file") && to_string($message.file) == "/var/log/audit/audit.log"
then
  let matches = grok("%{avc_extractor}", to_string($message.message));
  set_fields(matches);
end

Also, I should note that the wildly incorrect code I originally posted did not flag an error in the UI's syntax checker.

braiam commented 4 years ago

@joschi @jalogisch It doesn't produce an error anymore, but also doesn't produce a syntax error either.

Version: 3.2.4+a407287, codename Ethereal Elk JVM: Oracle Corporation 1.8.0_242 on Linux 4.9.0-12-amd64

rule "stunnel service message"
when
  has_field("application_name") 
    && to_string($message.application_name) == "stunnel"
    && NOT has_field("processed")
then
  let stunnel_message = to_string($message.message);
  let parsed_pattern = grok("STUNNELSERVICE", stunnel_message, true);
  set_fields(parsed_pattern);
  set_field("processed", true);
end

It only adds the processed field.

jalogisch commented 4 years ago

@braiam you have reverenced the Pattern the wrong way, that is why the pattern is not used to extract fields.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

braiam commented 4 years ago

@jalogisch I know I did it the wrong way, but @joschi said "This being said, a syntactically incorrect rule should never trigger a NullPointerException but a emit parse error instead". Graylog still doesn't emit a parse error and that feature is missing. The bug report isn't closed yet, so I presume that that's being still worked on, since the NPE has been addressed.

If not, this bug is already fixed. The wrong syntax don't trigger a NPE.

Graylog2 / graylog2-server