Adding some flexibility into Logstash

[GoogleCodeExporter]

If I could run grep or grok matching patterns and then if it matches I could 
send(output) again it to a AMQP queue, I could do a lot of stuffs doing this.

Some thoughts:
- I can do a post processing batch for some of my messages. In a heavy message 
input, I can choose which pattern messages I can index with elasticsearch right 
when I get the message and some I can send into another queue and index them 
later
- I can change the type using the "unique" queue that I sent the message
- Some pattern alerts, like if I get "fatal error" in the message I put them 
into a Alert queue and work on that right away or later...

It would be much better if I could choose the patterns and select outputs for 
each, like if I get pattern "fatal error" in the message I output it for 
elasticsearch and queue into a AMQP (+ output it) and a default output too.

More like integrating the outputs "modules" into filters

Original issue reported on code.google.com by luiz.oz...@gmail.com on 25 Apr 2011 at 11:19

[GoogleCodeExporter]

The grep filter will drop events that do not match. This allows you to 
selectively drop things that you don't want to output.

You can also add arbitrary tags to events using the grep filter.

The 'pattern alerts' would most likely be a grep that adds any necessary 
fields/tags, then an output that uses them. The nagios output is an example of 
this.

Currently, if you want to run multiple pipelines, you'll have to run multiple 
agents.

Original comment by jls.semi...@gmail.com on 26 Apr 2011 at 1:34

[GoogleCodeExporter]

I'd like to redirect some of the messages into a AMQP to parse them later, I 
don't want to drop them. Just analyse them later or do something else.

But yea, I was thinking in running multiples agents. The "master" listening 
syslog that would index my "real time" messages and re-queue others for post 
analysis into another queues.

And then other agents fetching that post queued messages for a night processing 
or giving me some alerts, for example.

The main idea is to make it more flexible, like if I could match pattern and 
send them to other queues, I could turn my "master" syslog listener agent like 
a router, so he just redirect messages to some queues and then a bunch of other 
agents processing them.

Just an idea...

Original comment by luiz.oz...@gmail.com on 26 Apr 2011 at 9:45