search

Graylog2 / graylog-plugin-cef

[DEPRECATED] CEF (Common Event Format) input plugin for Graylog
https://www.graylog.org/
Apache License 2.0
10 stars 10 forks source link

Message is not recognized as CEF and could not be parsed #19

Closed canob closed 6 years ago

canob commented 7 years ago

Hello,

I installed the CEF-plugin 1.2.0 using the RPM package on CentOS 7, restarted Graylog 2.2.3, and defined a new CEF input. However, no matter if I am using UDP or TCP, no logmessages are shown in Graylog2, although tcpdump is showing traffic, and netstat shows that the port is open.

I'm using ArcSight SmartConnector, version 7.6.0, colecting Unix logs, and I'm trying to send that logs to Graylog2, configuring as output "CEF Syslog" on ArcSight Connector.

In the graylog-server.log I can just see complains about the raw-message, but no other message regarding the CEF-plugin:

2017-07-18T16:01:19.334-03:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=7b981322-6beb-11e7-8e37-0050568ec090, journalOffset=239, codec=CEF, payloadSize=432, timestamp=2017-07-18T19:01:19.314Z, remoteAddress=/127.0.0.1:42636}
org.graylog.plugins.cef.parser.ParserException: This message was not recognized as CEF and could not be parsed.
        at org.graylog.plugins.cef.parser.SyslogCEFParser.parse(SyslogCEFParser.java:51) ~[?:?]
        at org.graylog.plugins.cef.codec.CEFCodec.decode(CEFCodec.java:59) ~[?:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:146) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:87) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:79) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:45) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

I'm think that there is no more "CEF compliance" than an ArcSight SmartConnector itself output, :'( , but maybe I'm not right.

This is what I'm seeing with tcpdump:

16:01:19.312101 IP 127.0.0.1.42636 > 127.0.0.1.5555: Flags [P.], seq 6630:7462, ack 1, win 342, options [nop,nop,TS val 19136937 ecr 18961919], length 832
E..t..@.@...............i.K     (T5....V.i.....
.$...!U.CEF:0|Unix|Unix||arcsight:143:1|Started Session|Low| eventId=31 msg=Started Session 21 of user root categorySignificance=/Informational categoryBehavior=/Access/Start categoryDeviceGroup=/Operating System catdt=Operating System categoryOutcome=/Success categoryObject=/Host/Application/Service art=1500404470493 deviceSeverity=info act=Started rt=1500404461000 suser=root dhost=centos7 cs1=systemd cs2=daemon cs1Label=Module cs2Label=Facility cn1Label=File Descriptor ahost=centos7.neolab agt=10.11.3.170 agentZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 amac=00-50-56-8E-C0-90 av=7.6.0.8009.0 atz=America/Argentina/Buenos_Aires at=syslog dvchost=centos7 dtz=America/Argentina/Buenos_Aires deviceFacility=daemon deviceProcessName=systemd _cefVer=0.1 aid=3SNQJV10BABCAIi+-ZH5gxQ\=\=

All the resources (ArcSight Connector, Syslog Server, Graylog Server) are on the same machine, because this is a lab environment.

Can you help us, or point us in the right direction to solve this issue?

Thanks for your help.

Regards, Alejandro Guida

joschi commented 7 years ago

@canob The CEF plugin currently expects the messages to be received within a syslog message.

From the tcpdump output you've attached it looks like your clients are sending the CEF payload directly and not within a syslog message.

As a workaround you can create a Raw/Plaintext input and use the parse_cef() pipeline function provided by this plugin to extract the information from these messages.

canob commented 7 years ago

Thanks @joschi

I already tried the workaround that you mentioned with no luck, because in the CEF syslog that I'm receiving, the Severity is coming in text as the parser expected, not numeric: CEF:0|Unix|Unix||arcsight:143:1|Started Session|Low|

There is a fork that solved this problem, but it is based on an old version of this plugin, and that fork don't has parse_cef() pipeline function, :-(

In this moment ArcSight compatibility is a must have for us, so I'm going to try ELK stack, which have ArcSight Compatibility natively with a new plugin that come with X-Pack.

Thanks for your help.

joschi commented 7 years ago

@canob We'll put a bit of effort into this plugin over the next few days.

Maybe you want to try the new release (when it's out).

joschi commented 7 years ago

@canob Please give the latest version of the plugin a try (depending on your Graylog version):

You might have to re-create your CEF input depending on your message sources, e. g. a "CEF TCP/UDP input" for ArcSight instead of a "CEF Syslog TCP/UDP input".

canob commented 7 years ago

Hi @joschi

I tried 2.3.0-beta.1, but after install the plugin, graylog-server service can't start, and I'm seeing this message on server.log:

2017-09-19T14:05:23.827-03:00 INFO  [CmdLineTool] Loaded plugin: Delimited File Writer 0.1.0 [com.rswestmoreland.graylog2.plugin.DelimitedFileOutput]
2017-09-19T14:05:23.829-03:00 INFO  [CmdLineTool] Loaded plugin: SyslogOutputPlugin 1.0.0 [com.wizecore.graylog2.plugin.SyslogOutput]
2017-09-19T14:05:23.830-03:00 INFO  [CmdLineTool] Loaded plugin: Aggregates 1.1.1 [org.graylog.plugins.aggregates.AggregatesPlugin]
2017-09-19T14:05:23.831-03:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.3.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-09-19T14:05:23.832-03:00 INFO  [CmdLineTool] Loaded plugin: CEF Input 2.3.0-beta.1 [org.graylog.plugins.cef.CEFInputPlugin]
2017-09-19T14:05:23.832-03:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.3.1 [org.graylog.plugins.collector.CollectorPlugin]
2017-09-19T14:05:23.833-03:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.3.1 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-09-19T14:05:23.834-03:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.3.1 [org.graylog.plugins.map.MapWidgetPlugin]
2017-09-19T14:05:23.845-03:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.3.1 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-09-19T14:05:23.846-03:00 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.3.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-09-19T14:05:24.163-03:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2017-09-19T14:05:24.429-03:00 INFO  [Version] HV000001: Hibernate Validator null

After I removed the .jar file of cef plugin, graylog-server service starts without any problem.

Thanks for all your help.

joschi commented 7 years ago

@canob This has been fixed in version 2.3.0-beta.2.

canob commented 7 years ago

@joschi

Now the plugin installed ok and Graylog Server is UP and running, but when I tried to create a CEF Kafka Input, I received this error in the Gui:

Could not launch input Launching input 'CEF Kafka Input' failed with status: Error: cannot POST http://10.11.3.170:900/api/system/inputs (400)

In server.log file, I can see this error when I try to create the Input:

2017-09-20T07:38:59.074-03:00 ERROR [InputsResource] Missing or invalid input configuration.
org.graylog2.plugin.configuration.ConfigurationException: Mandatory configuration field bind_address is missing or has the wrong data type
        at org.graylog2.plugin.configuration.ConfigurationRequest.check(ConfigurationRequest.java:117) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.checkConfiguration(MessageInput.java:145) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.inputs.InputsResource.create(InputsResource.java:133) [graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_144]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_144]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_144]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_144]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_144]

Thanks for all your help.

canob commented 7 years ago

@joschi

Trying the parse_cef function (which is part of the plugin) with a Syslog UDP Input, , I found that the CEF messsage is not being parsed correctly, :'(

I'm using this function because I'm receiving CEF from an ArcSight Connector (so I know the "CEF text"is correctly constructed), sending Syslog CEF, which not have Syslog Header, only the "CEF text".

The CEF message:

CEF:0|Unix|Unix||arcsight:143:22|Created slice|Low| eventId=212113 msg=Created slice user-994.slice catdt=Operating System art=1505922484252 deviceSeverity=info act=Created rt=1505922481000 dhost=centos7 dst=10.11.3.170 destinationZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 cs1=systemd cs2=daemon cs1Label=Module cs2Label=Facility cn1Label=File Descriptor ahost=centos7 agt=10.11.3.170 agentZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 amac=00-50-56-8E-C0-90 av=7.6.0.8009.0 atz=America/Argentina/Buenos_Aires at=syslog dvchost=centos7 dvc=10.11.3.170 deviceZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 dtz=America/Argentina/Buenos_Aires deviceFacility=daemon deviceProcessName=systemd _cefVer=0.1 ad.slice=user-994.slice aid=3SNQJV10BABCAIi+-ZH5gxQ\=\=

The error in log:

2017-09-20T12:54:27.582-03:00 ERROR [CEFParserFunction] Could not run CEF parser for [message].
org.graylog.plugins.cef.parser.ParserException: CEF pattern did not match. Skipping this message.
        at org.graylog.plugins.cef.parser.CEFParser.parse(CEFParser.java:65) ~[graylog-plugin-cef-2.3.0-beta.2.jar:?]
        at org.graylog.plugins.cef.pipelines.rules.CEFParserFunction.evaluate(CEFParserFunction.java:56) [graylog-plugin-cef-2.3.0-beta.2.jar:?]
        at org.graylog.plugins.cef.pipelines.rules.CEFParserFunction.evaluate(CEFParserFunction.java:22) [graylog-plugin-cef-2.3.0-beta.2.jar:?]
        at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.evaluateUnsafe(FunctionExpression.java:63) [graylog-plugin-pipeline-processor-2.3.1.jar:?]
        at org.graylog.plugins.pipelineprocessor.ast.expressions.Expression.evaluate(Expression.java:41) [graylog-plugin-pipeline-processor-2.3.1.jar:?]
        at org.graylog.plugins.pipelineprocessor.ast.statements.FunctionStatement.evaluate(FunctionStatement.java:32) [graylog-plugin-pipeline-processor-2.3.1.jar:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.evaluateStatement(PipelineInterpreter.java:377) [graylog-plugin-pipeline-processor-2.3.1.jar:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.executeRuleActions(PipelineInterpreter.java:364) [graylog-plugin-pipeline-processor-2.3.1.jar:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.evaluateStage(PipelineInterpreter.java:305) [graylog-plugin-pipeline-processor-2.3.1.jar:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.processForResolvedPipelines(PipelineInterpreter.java:263) [graylog-plugin-pipeline-processor-2.3.1.jar:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.process(PipelineInterpreter.java:143) [graylog-plugin-pipeline-processor-2.3.1.jar:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.process(PipelineInterpreter.java:99) [graylog-plugin-pipeline-processor-2.3.1.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:114) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:100) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:77) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_144]

Thanks for all your help.

HernanMora commented 7 years ago

Hello @joschi @canob and I are also having some issues in the way the plugin parses CEF messages, all related with CEF format:

a) CEF prefix fields (Fields between pipes): Some devices don't report their device version, joining two pipes together. Regexp requires all values between pipes, otherwise the parse fails.

b) An event can contain any number of key-value pairs in any order, separated by spaces (“ “). So, when key-value msg comes at the beginning, all key-value pairs which come after it are encapsulated as part of msg value.

c) We are receiving CEF messages with and without Syslog transport. We need that plugin supports both message types at the same time (Regexp expects Syslog header)

You can use something like this for key-value pairs:

   `Matcher m = Pattern.compile("(?<!\\\\)=").matcher(extension);

    // Parse out all fields into a map.
    ImmutableMap.Builder<String, String> fieldsBuilder = new ImmutableMap.Builder<>();

    if (extension != null && !extension.isEmpty()) {

        int index = 0;
        String key = null;
        String value = null;
        while (m.find()) {
            if (key == null) {
                key = extension.substring(index, m.start());
                index = m.end();
                if (!m.find()) {
                    break;
                }
            }
            value = extension.substring(index, m.start());
            index = m.end();
            int v = value.lastIndexOf(" ");
            if (v > 0) {
                String temp = value.substring(0, v).trim();
                fieldsBuilder.put(key, String.valueOf(temp));
                // Next Key
                key = value.substring(v).trim();
            }

        }
        // Value of last key
        value = extension.substring(index);
        fieldsBuilder.put(key, String.valueOf(value));
    }`

Thanks in advance!

joschi commented 7 years ago

@HernanMora

CEF prefix fields (Fields between pipes): Some devices don't report their device version, joining two pipes together. Regexp requires all values between pipes, otherwise the parse fails.

Do you have an example for these devices and some example messages?

An event can contain any number of key-value pairs in any order, separated by spaces (“ “). So, when key-value msg comes at the beginning, all key-value pairs which come after it are encapsulated as part of msg value.

This seems to depend on the actual CEF producer implementation, for example OSSEC produces messages like this (with a trailing msg field without properly escaped value):

Nov  6 13:16:03 ossecsrv CEF:0|Trend Micro Inc.|OSSEC HIDS|v2.7|5402|Successful sudo to ROOT executed|3|dvc=ossecsrv cs2=(ossecclient) 10.64.11.188->/var/log/messages cs2Label=Location suser=nagios msg=Nov  6 13:16:02 ossecclient sudo:   nagios : TTY=unknown ; PWD=/opt/home/nagios ; USER=root ; COMMAND=/bin/cat /proc/linuxshield/enabled
HernanMora commented 7 years ago

@joschi

We are using ArcSight connectors. Here is a CEF message example from Unix SSH connection:

CEF:0|Unix|Unix||arcsight:143:1|Started Session|Low| eventId=220687 msg=Started Session 24116 of user root categorySignificance=/Informational categoryBehavior=/Access/Start categoryDeviceGroup=/Operating System catdt=Operating System categoryOutcome=/Success categoryObject=/Host/Application/Service art=1505992554252 deviceSeverity=info act=Started rt=1505992552000 suser=root dhost=centos7 dst=10.11.3.170 destinationZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 cs1=systemd cs2=daemon cs1Label=Module cs2Label=Facility cn1Label=File Descriptor ahost=centos7 agt=10.11.3.170 agentZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 amac=00-50-56-8E-C0-90 av=7.6.0.8009.0 atz=America/Argentina/Buenos_Aires at=syslog dvchost=centos7 dvc=10.11.3.170 deviceZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 dtz=America/Argentina/Buenos_Aires deviceFacility=daemon deviceProcessName=systemd _cefVer=0.1 aid=3saYCNV4BABCAOKKeaywhGg\=\=

If you want to see how ArcSight defines its format, you can check this link: ArcSight CEF.

Thanks for your time!

joschi commented 7 years ago

Next plugin iteration:

joschi commented 7 years ago

And even more (only fixes AMQP and Kafka CEF inputs):

canob commented 7 years ago

Hi @joschi,

First of all, thanks for all your hard work in this topic, is really amazing all the changes that you did the last days.

After we tried 2.3.0-beta.4, version, now we have the fields correctly mapped.

The new issues that we found are:

Again, thanks for your help and your time.

joschi commented 7 years ago

the fields corresponding to IPv4 addresses, like deviceAddress, sourceAddress, or destinationAddress, are not being mapped into ElasticSearch as "ip"

Correct. You need to create a custom index mapping for that: http://docs.graylog.org/en/2.3/pages/configuration/elasticsearch.html#custom-index-mappings

(Minor): the fields corresponding to CEF Header (deviceVendor, deviceProduct, etc.) are not named correctly according to CEF standard. For example, deviceVendor is named as device_vendor.

Fair enough. We might change that in one of the next releases.

In the meantime, you can use processing pipeline rules to rename existing fields: rename_field()

joschi commented 6 years ago

The remaining issues with the CEF plugin (such as not parsing CEF messages sent by OSSEC via syslog correctly) should be solved with the latest releases for Graylog 2.2.x and 2.3.x respectively:

If you still have issues with the plugin, please open a new issue and include sample messages so that we can reproduce the problem.