search

Graylog2 / graylog-plugin-pipeline-processor

[DEPRECATED] Graylog Pipeline Message Processor Plugins
https://www.graylog.org/
GNU General Public License v3.0
20 stars 15 forks source link

Problem like #24 #211

Closed akiontke closed 6 years ago

akiontke commented 6 years ago

Problem description

Can't save a rule with ( in grok pattern

Steps to reproduce the problem

  1. Create Rule
rule "extract_mac"
when
  contains(value: to_string($message.full_message), search: "DHCPREQUEST")
then
    let matches = grok(pattern: "DHCPREQUEST for %{IPV4:client_ip} from %{COMMONMAC:client_mac} \\(%{NOTSPACE:client_name}\\)", value: to_string($message.full_message));
    set_fields(matches);
end
  1. Try to save

Environment

joschi commented 6 years ago

@akiontke Please check the logs of your Graylog node(s) and attach the complete error messages which occur when trying to save the rule.

akiontke commented 6 years ago

@joschi I get the following error message with the mentioned example. 

2017-09-28T10:56:33.600+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.NullPointerException: null
        at org.graylog.plugins.pipelineprocessor.ast.functions.FunctionArgs.lambda$getConstantArgs$0(FunctionArgs.java:57) ~[?:?]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[?:1.8.0_102]
        at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1691) ~[?:1.8.0_102]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_102]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_102]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_102]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_102]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_102]
        at org.graylog.plugins.pipelineprocessor.ast.functions.FunctionArgs.getConstantArgs(FunctionArgs.java:59) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.ast.functions.Function.preprocessArgs(Function.java:54) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.<init>(FunctionExpression.java:44) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser$RuleAstBuilder.exitFunctionCall(PipelineRuleParser.java:411) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.RuleLangParser$FunctionCallContext.exitRule(RuleLangParser.java:1434) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:170) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:135) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.rest.RuleResource.update(RuleResource.java:174) ~[?:?]
        at sun.reflect.GeneratedMethodAccessor415.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
joschi commented 6 years ago

@akiontke I just tried to reproduce this issue on Graylog 2.3.1 (Docker image) but everything worked as expected.

create_rule_1 create_rule_2

Please post the contents of the System / Nodes / Details page ("Installed plugins" specifically). Maybe you're running an old/incompatible version of the Pipeline Processor Plugin?

Also check the contents of the System / Grok Patterns page and make sure that all referenced Grok patterns in your rule (IPV4, COMMONMAC, etc.) do exist.

For reference: docker-compose.yml

version: '2'
services:
  mongodb:
    image: mongo:3
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.2
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  graylog:
    image: graylog/graylog:2.3.1-2
    mem_limit: 4g
    environment:
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      - 9000:9000
akiontke commented 6 years ago

These are the plugins in use

Name Version Author Description
Anonymous Usage Statistics 2.3.1 Graylog, Inc. A plugin for collecting anonymous usages statistics about Graylog nodes and clusters.  Website
Collector 2.3.1 Graylog, Inc. Collectors plugin  Website
Elastic Beats Input 2.3.1 Graylog, Inc. Input plugin for Elastic Beats (Beats/Lumberjack protocol).  Website
Enterprise Integration Plugin 2.3.1 Graylog, Inc Provides basic integration with Graylog Enterprise  Website
Internal Metrics InfluxDB Reporter 1.4.0 Graylog, Inc. A plugin for reporting internal Graylog metrics to InfluxDB.  Website
MapWidgetPlugin 2.3.1 Graylog, Inc. Map widget for Graylog  Website
NetFlow Plugin 2.3.0-rc.5 Graylog, Inc. Provides NetFlow inputs  Website
Pipeline Processor Plugin 2.3.1 Graylog, Inc Pluggable pipeline processing framework  Website
Slack 2.4.0 Graylog, Inc. Slack plugin to forward messages or write alarms to Slack chat rooms.  Website

All mentioned grok patterns are configured