Allow pipeline stage to connect another pipeline

[jasonkeller]

I don't see this in my Github searches, so i figured I'd ask here. I ran into an issue with slookup (https://github.com/billmurrin/graylog-plugin-slookup-function/issues/9) for a specific pipeline rule that consistently failed to work (and dropped messages out of that stream to boot).

I had to work around this by separating these two when clauses into different stages, and place them into a completely separate pipeline. Due to other constraints, I had to also create a different stream to attach this pipeline to to ensure that the parsing was done before these alert condition rules.

It would be more flexible for complex conditions like these to be able to attach another pipeline to start at a specific stage of a processing pipeline rather than having to tag it into another stream to accomplish the same. Or perhaps a route_to_pipeline function that additionally begins processing the message through another pipeline (while finishing the original pipeline it's currently processing through, or even better a flag whether to continue on the original while sending it down the new pipeline or cut the original pipeline short).

[joschi]

@jasonkeller Please provide some example illustrating your issue.

I have a hard time understanding what you're trying to achieve, to be honest.

[jasonkeller]

@joschi Sorry, morning ramble.

So I had a request from our SOC to alert on specific messages in a stream (Palo Alto Traps) where we have multiple messages come in with the same value in the 'filehash' field, but with different values in the 'destinationHostName' field. Obviously this cannot be accomplished using standard alerting functionality in Graylog, hence I need to use pipelines to selectively create the complex criteria and enrich the log with another field that the alert mechanism can match on. It also requires being able to query fields from previous messages, hence my use of @billmurrin slookup() function plugin.

This led me to my second dilemma - placing these two slookup queries in a single pipeline when clause not only wouldn't work, but crashed the pipeline processor badly enough where it was consistently dropping messages entirely. What I ended up having to do to solve this was to separate both of these calls into two different pipeline rules and stages, and placing it in an independent pipeline.

This leads to a third dilemma - due to the Palo Alto Traps team's seeming lack of knowledge of RFC 3164 syslog headers (https://github.com/Graylog2/graylog-plugin-cef/issues/26) I cannot use the CEF plugin, therefore in the first pipeline stage I must parse the message using the parse_cef function. Unfortunately this means there is no way for me to require this pipeline to process first, then process the other pipelines (they will run in parallel). Since the fields don't exist yet prior to the pipeline stage that parses in one pipeline, the second pipeline will not have the requisite fields necessary to complete evaluation of the message.

Which all comes down to my kludge - create one stream that leads into the pipeline that parses, then create another stream that the message gets tagged into (via route_to_stream at the end of the parsing rule) which the two alerting pipelines attach to (one for the simpler alert conditions, the other with this complex multi-stage conditional). It would have been significantly easier to be able to route a message into an additional pipeline at a certain stage without creating yet another stream as an intermediary.