Drools applies not to all messages, if more than 1 processor enabled

[hc4]

Problem description

If I set processbuffer_processors to more than 1, Drools will process not every message

Steps to reproduce the problem

1. Set processbuffer_processors in server.conf to 3 (for example)
2. Create simple Drools rule, which will just create new field for every message: rule "Test Rule" when m : Message() then m.addField("test", "test_value"); end
3. Push some messages
4. Not all messages will have new field

   Environment

   • Graylog Version:2.0 beta3
   • Elasticsearch Version:2.3.1

[kroepke]

I cannot reproduce this issue, just tried with exactly that setup on current master and everything looks fine. Are there any exceptions in your log?

[hc4]

No errors in log. Maybe the problem is in BeatsInput plugin, which I'm currently using. Also I use TopBeat as data source (it generates hundreds of messages in single packet) Also I have one extractor, which I can not edit - will try to remove it and reproduce problem.

[hc4]

Removed all extractors and disconnected all pipelines. Still no luck. I've attached csv export of saved messages. Time is differend for processed and unprocessed messages, because I also change timestamp by Drools (actually it could be problem maybe)

graylog-search-result-relative-300.txt

[kroepke]

The input plugin should not have anything to do with it, however, it is also not built for Graylog 2.0 yet, so there might be incompatibilities.

Can you try to reproduce this issue with the random http message generator input?

[hc4]

Checked with random HTTP generator and it works. Maybe the problem with BeatsTransport?

[kroepke]

Ok, then it sounds like there's an issue with the beat input.

It could very well be that some fields have a . in them? Those are no longer compatible with elasticsearch and the Graylog behavior changed after beta.3, iirc: https://github.com/Graylog2/graylog2-server/pull/2078/commits/3953af2f755200e59adfcd26986cbeba4aab457e

[hc4]

I fixed that by changing all dots to "_" in input. Is it possible, that Input incorrectly pushes messages to queue?

[kroepke]

All messages from an input are processed in the same manner, I don't know the beats input plugin that well, but there's usually no way to avoid processing them. With the _ is the issue resolved?

[hc4]

No. I've faced dot problem on first Graylog 2.0 start and fixed it in first place

[hc4]

created log with trace level while reproducing problem (GA version still have this bug) Also attached CSV export. Rows with topbeat_proc_cpu_system field are bad (don't know how to export GUID to CSV)

gaylog_bug_trace.txt gaylog_bug_data.txt

For example. good message: ad48ae13-121a-11e6-b5bc-005056b37533 bad message: ad3eea10-121a-11e6-b5bc-005056b37533

[hc4]

Reproduced with Random HTML generator

Generator settings:

sleep: 1
override_source:
source: example.org
sleep_deviation: 2
throttling_allowed:

Rule:

rule "Test Rule"
    when
        m : Message( true )
    then
        m.addField("test", "1");
end

[ghost]

I can also confirm that drool rules are not applied to all messages in Graylog 2.0.0.

• Graylog server 2.0.0 (2dc6c03)
• The issue did not exist on 1.x.
• Both SyslogUDP and GELF UDP inputs are affected (that's all of the inputs we're using Drool rules with)
• We do not use the pipeline processor. The system was just upgraded from a 1.x install.
• @hc4 's observation that setting processbuffer_processors=1 works around the issue is correct.

[kroepke]

I've ran a longer test to try to reproduce this and once in a while it looks like Drools itself is not matching any rules:

2016-05-13 13:00:47,581 ERROR: org.graylog2.rules.DroolsRulesSession - Should have fired rules! Thread 30

This happens across all processor threads (I am running three in this case), without an obvious pattern.

The weird thing about this issue is, that nothing about Drools has changed since 1.2, in fact nothing has changed since July 2014. I'll keep digging, though.

[hc4]

Could you check if RulesFilter are singleton, or created per processor thread? (it is hard for me to check without IDE ) I think, that RulesFilter is not thread safe and must be called only from single thread (because Drools are not thread safe itself).

Also i found, that extractors processed before drools according to that code:

        Multibinder<MessageFilter> messageFilters = Multibinder.newSetBinder(binder(), MessageFilter.class);
        messageFilters.addBinding().to(StaticFieldFilter.class);
        messageFilters.addBinding().to(ExtractorFilter.class);
        messageFilters.addBinding().to(RulesFilter.class);
        messageFilters.addBinding().to(StreamMatcherFilter.class);

Could it be made configurable throught "Message Processors Configuration" page?

[kroepke]

I have found the issue and will push a fix shortly. It is because the OrderedMessageProcessors was a singleton, but shouldn't be. Previously each thread was injected its own copy, which is how it was supposed to, but afterwards, the instances were shared.

All filters can cope with that, except Drools. Its sessions need to be isolated.

We will not be making the filters configurable, because that interface will be deprecated soon. Instead the Drools support will move into a plugin in an upcoming version and implement MessageProcessor instead.

[kroepke]

Also to avoid confusion, the binding code quoted above does not influence the execution order, filters are ordered by a numeric priority which is hardcoded.