EqualityExpression: left expression evaluated to null

[tokred]

Problem description

After adding a rule, I am receiving dozends of Graylog log warnings

[EqualityExpression] left expression evaluated to null, returning false: $message.log_type

However, the field log_type should never be empty in my messages as it is set on Logstash collectors. Graylog gives me no results when I check for messages withouth the field log_type via a query _missing_:log_type.

Warnings stop if I add a has_field() check to the rule:

rule "Log type: log-alu-timos"
when
  has_field("log_type") && ($message.log_type == "log-alu-timos")
then
  route_to_stream(id: "5731f3a4fa16656c2c26ee1a");
end

Where do these warnings come from?

Steps to reproduce the problem

unclear, please advise

Environment

• Graylog Version: 2.0.0
• Pipeline Processor plugin version: 1.0.0-beta.2
• Elasticsearch Version: 2.3.1
• MongoDB Version:
• Operating System:
• Browser version:

[kroepke]

@tokred The $message.log_type should only be null if the field does not exist. The code basically just calls message.getField("log_type") which is only null if the field wasn't set.

Did you search across all the messages, or just in the referenced stream?

We've made the message verbose on purpose because this failure usually indicates a wrong assumption about how the data looks like.

[tokred]

I searched across all messages for the time frame where I received the EqualityExpression warnings and got no hits for messages that lack the referenced field. Maybe those got lost and it's indication of some bug? Impossible to reproduce though, I guess :(

[kroepke]

That might be true, of course.

In any case the warning simply means that this rule did not apply, but the message should've gone through. Let's keep this open to try to reproduce it later.

[tokred]

tl;dr: solved, I think Did some further analysis. Our setup consists of a Kafka-based message queue which spools all messages in different topics. Graylog connects to Kafka on several inputs (=topics) to consume messages.

After changing the pipeline rule back to the original state (no extra has_field check), I made a full capture of all messages for a few minutes using the Kafka console consumer on all topics. During this time, I received approx. 20 EqualityEpression warnings in Graylog logs for 1570 messages that went through the message queue:

$ wc -l testdump2.kafka
1570 testdump2.kafka

$ grep -o -P '"log_type":".*?"' testdump2.kafka | sort | uniq -c
     50 "log_type":"log-alu-timos"
   1474 "log_type":"log-cisco-asa"
     33 "log_type":"log-generic"
     10 "log_type":"log-heartbeat"
      3 "log_type":"log-juniper-junos"

All of the 1570 messages contain a non-null value for field "log_type", although I got Graylog warnings.

Update: I found the root cause - the order of Pipeline Processor and Message Filter Chain! I have an additional local input for Graylog internal logs which adds a static field for "log_type". However, this is applied AFTER pipeline processing, therefore those messages indeed had no log_type field while processing the rule, but still had it in Elasticsearch.

[kroepke]

Thank you very much for the update!