Streams - Message without field + "field must not match" > won't match

[SjonHortensius]

I have a stream rule: "request must not match regular expression ^OPTIONS /". Any message that doesn't have a "request" field won't end up in this stream. Isn't that a bug?

[kroepke]

It does sound odd. But one can argue either way, I guess.

Maybe the correct way of describing this would be with an additional checkbox for what to do if the field is not present?

[SjonHortensius]

Well the additional checkbox would definitely fix this. Personally I still think the message should match when the field isn't present, but I guess it isn't straight forward

[tillo]

I am having this problem exactly. I don't want to match when a field has some content, but I want to match when the same field is absent.

In my opinion it is an important bug, at least until an OR feature is included, because the only workaround has a significant performance hit: I can bypass the field completely and inversely match a regexp containing the same extractor regexp that creates the field, plus the desired content. This means that the regexp for the field will run twice.

Consider for example the Linux Auditd extractors in the Marketplace, where field audit_exe is filled thanks to the exe="(.+)" regexp. I cannot add a stream rule for !audit_exe:"/usr/sbin/cron", because not every message has the exe= part, and those will also be excluded. Obviously, that's not what I want: if I wanted that, I would have added _exists_:audit_exe to the rules. If there was an OR feature, I could fix the situation with OR !_exists_:audit_exe, but all the rules are implicit AND. Instead, I had to use the rule !message:exe="/usr/sbin/cron" which will search in the entire message again and is therefore inefficient.

This is unsolved since 2014 and I thought it deserved a reminder. :)

[edmundoa]

@tillo For what it's worth, the stream rule OR operator was released in 1.2.

[tillo]

@edmundoa I'm not sure if you mean the "A message must match at least one of the following rules" option. If you do, I would have swapped a problem with another. If I can choose whether a stream rule is AND or OR (and possibly use parenthesis/groups for more complex combinations), please help me understand how, the documentation does not mention it.

[edmundoa]

@tillo selecting "A message must match at least one of the following rules" in your stream rules configuration will do an OR with all the stream rules, so a message matching one of the rules will be routed into the stream. You are right regarding documentation, I created an issue to fix that: https://github.com/Graylog2/documentation/issues/52

[tillo]

@edmundoa Since that's what you meant, I will repeat what I wrote before: this may solve this problem, but it creates another, because I want other rules to be AND. Only a global AND behavior with per-rule OR feature would let me do what I want to do. But that's another bug report completely, one that is already open, by the way.

On this bug report, I am arguing that excluding messages without a certain field, when excluding some of that field content, is the wrong choice.

[DenEwout]

What is the current status on this? I'm having the exact same issue. I don't want messages to be excluded due to a field not being present when there is a rule that excludes based on that field's content.

[mikkolehtisalo]

@DenEwout I recently had a similar issue, and used pipelines to process the messages, and add a new boolean field as result. The rules are a bit nasty, but seemed to work well enough for me.