pipeline is not working, but simulator show result

[jalogisch]

Expected Behavior

After Update from 2.1.beta.1 i would expect that all pipelines just continues to work as before.

This is the first rule in a pipeline that needs to be true to start extracting content from the message

rule "extract if ufw"
when
 has_field("programname") AND to_string($message.programname) == "kernel"
then
 let message_field = to_string($message.message);

 let action = regex("^\\[((?:[0-9]+)).((?:[0-9]+))\\] \\[UFW.*", message_field);
 set_field("ufw", action.matches);

end

Current Behavior

The messages remain unprocessed, because this first rule did not match.

take the same message to the simulator just fire up the pipeline

Steps to Reproduce (for bugs)

Take this message as GELF Input in the Simulator

{"@timestamp":"2016-08-04T04:31:03.528Z","@version":"1","message":"[10861026.240439] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=104.193.252.230 DST=91.250.117.97 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=46901 DPT=21320 WINDOW=65535 RES=0x00 SYN URGP=0 ","host":"box.jalogis.ch","severity":"warning","facility":"kern","programname":"kernel","procid":"-","type":"syslog"}

1. create a pipeline with the rule in the description
2. send the provided message to graylog input

   Your Environment

   • Graylog Version: 2.1. beta 2 (OVA/Omnibus)

[edmundoa]

The example message fulfils the condition has_field("programname") AND to_string($message.programname) == "kernel", so I don't think this is a problem with the simulator. If the messages are left unprocessed, could you check if the pipeline is attached to the right stream?

[jalogisch]

This was just right after the update from beta 1 (where it is working) to beta 2 (where it is nor working) - no other change done.

[jalogisch]

Just to clarify how the processing timeline is in this setup:

so before this mentioned rule only this rule fired up:

rule "drop message gelf set true"
when
 has_field("from_gelf") AND  to_bool($message.from_gelf) == true
then
  drop_message();
end

[kroepke]

I cannot reproduce this issue. I've set up the given rule and connected it to the default stream. Sending the message to a GELF input (even though it is not valid GELF!) causes the message to be processed on current master Graylog 2.1.0-beta.3-SNAPSHOT https://github.com/Graylog2/graylog2-server/commits/60deda2

Please reopen this issue if there is additional information.

[jancaster]

Just in case someone ran into this issue, I have experienced the same on Graylog 2.1.1+01d50e5, and elasticsearch is throwing indexing failures "MapperParsingException"s, In my case it was a type mismatch, I had erroneously indexed an {object} field, then tried to index strings on the same field after correcting the pipeline rules.

I was able to clear the issue by manually cycling the deflector index.

The simulator would show the expected behaviour since it does not go through the indexes/storage

[achevalet]

I've the same issue with 2.1 stable - but only with custom stream. With the default stream the rule is correctly applied. With custom stream the rule is NOT applied.

[mwysokin]

I'm on Graylog 2.2.3+7adc951 (Oracle Corporation 1.8.0_72-internal on Linux 4.4.0-81-generic) and see the same thing as @achevalet . Pipelines receive messages only from the default stream. When a custom one is connected even though it gets messages they are not supplied to the pipeline.

[joschi]

@mwysokin Please create a new issue for this and include your pipeline rules and some example messages, so we can try to reproduce the issue.

[louzbeck]

Just a +1 to @achevalet and @mwysokin 's comments.

Currently running Graylog 2.3.1+9f2c6ef (Oracle Corporation 1.8.0_144 on Linux 4.4.0-96-generic)

I'm experiencing the same behavior, a pipeline with a very basic rule:

rule "discard snmp daemon noise"
when
    contains(to_string($message.message),"Connection from UDP")
then
    drop_message();
end

=> does nothing when pipeline is set to process messages from my rsyslog stream (although tests in simulator succeed) => works just fine when pipeline is set to process message from "All messages" stream.

(and so for now managed to get the desired behavior by unchecking option "Remove matches from 'All messages' stream" in my rsyslog stream config)

[joschi]

@louzbeck Please provide a sample message which should be matched by this rule. Otherwise we'll be unable to reproduce your problem.

[louzbeck]

@joschi Have re-created the pipeline as i had originally created it (so set to process message from "rsyslog" stream, not the default "All messages" one)

Raw message example: <30>Oct 4 20:27:04 xxxxxxxx snmpd[2990]: Connection from UDP: [127.0.0.1]:38029->[127.0.0.1]:161 (message codec: Syslog)

Simulator says:

Simulation results

These are the results of processing the loaded message. Processing took 327 µs.

0 μs Starting message processing 71 μs Message 93fcea80-a9da-11e7-9845-00163e0709ef running [Pipeline 'RSYSLOG-TEST' (59d640d6f778063664649685)] for streams [59d2a50bb8a7772bb0d2c47f] 205 μs Enter Stage 0 246 μs Evaluate Rule 'discard SNMP daemon noise' (59d2ad06aeb740338067a139) in Pipeline 'RSYSLOG-TEST' (59d640d6f778063664649685) 275 μs Evaluation satisfied Rule 'discard SNMP daemon noise' (59d2ad06aeb740338067a139) in Pipeline 'RSYSLOG-TEST' (59d640d6f778063664649685) 307 μs Execute Rule 'discard SNMP daemon noise' (59d2ad06aeb740338067a139) in Pipeline 'RSYSLOG-TEST' (59d640d6f778063664649685) 311 μs Completed Stage 0 for Pipeline 'RSYSLOG-TEST' (59d640d6f778063664649685), continuing to next stage 313 μs Exit Stage 0 327 μs Finished message processing

Changes in original message 93fcea80-a9da-11e7-9845-00163e0709ef Original message would be dropped during processing.

Yet those messages don't get dropped unless i do what i mentioned in my previous comment.

To be more precise, it seems i got misled into thinking i needed to do:

unchecking option "Remove matches from 'All messages' stream" in my rsyslog stream config

I've just double checked, it does not seem to make any difference whether or not i set this option in the rsyslog stream config. It seems that the pipeline connected to "All messages" is applying the "discard SNMP daemon noise" rule before rsyslog messages get routed to the rsyslog stream.

What seems to really matter to get that pipeline/rule to work is that it has to be set to process messages from the "All messages" stream (and not some custom one).

[joschi]

@louzbeck What's the order of message processors in your Graylog setup?

You can find this information on the System / Configurations page in the "Message Processors Configuration" section.

Also, what's the exact pipeline configuration (stage order, rules order) you're using?

[louzbeck]

@joschi

Pipeline config is very basic (one stage containing the rule i mentioned before)

System / Configurations page in the "Message Processors Configuration":

#	Name	Active
1	GeoIP Resolver	active
2	Pipeline Processor	active
3	Message Filter Chain	active

Must admit i've never looked closely at this until now.. which got me going to http://docs.graylog.org/en/2.3/pages/pipelines/usage.html and read:

In that screen, you need to enable the Pipeline Processor message processor, and you must also set the Pipeline Processor after the Message Filter Chain, if you want to be able to use pipelines in your streams [..]

And so i've now got the unconfortable feelling that this is probably just another of those RTFM case :P

Will put "Pipeline Processor" after "Message Filter Chain" now, check again and let you know.

EDIT: yep, i confirm, doing as instructed in the documentation seems to be working just fine, very sorry for the noise here.

[joschi]

@louzbeck No worries. Thanks for your feedback!