Conditioned Pipelines additional to filters?

[ghost]

Idea: What about using Conditioned Pipelines additional to filters? This might solve my requirement stated in https://github.com/certtools/intelmq/issues/569#issuecomment-233544614

Each destination pipeline has an entry-condition which has to be met before an event is inserted into the pipeline. Maybe you can imagine it like a bouncer in front of a club. The default condition for each pipeline is true, so every event can get into the pipeline.

How might this look in a pipeline.conf file?

    "parser": {
        "source-queue": "parser-queue",
        "destination-queues": {
            "expert1-queue": "if event.get(\"source.cc\") = \"DE\""
            "expert2-queue": "if event.get(\"source.cc\") != \"DE\""
        }
    },
    "expert2": {
        "source-queue": "expert2-queue",
        "destination-queues": {
            "another-expert-queue": true
            "yet-another-expert-queue": true
        }
    },

Problems:

1. This allows the execution of arbitrary code from the *.conf files, if a programming language like python is used instead of a rule-language.

Questions:

1. What happens to bounced events?

Originally posted by @dmth in https://github.com/certtools/intelmq/issues/569#issuecomment-282973522

[ghost]

In order to not use python code in the configuration we could use the sieve's bot syntax and it's capabilities

[e3rd]

That would effectively solve use case of MaltaCIP. :) They have a file collector that receives all shadowserver files in a directory whose contents should be divided into shadowserver parsers. I advised to use a sieve bot that will distribute events to according parsers. But as an expert, sieve cannot be placed between collector and parsers.

(Could you please remind me of the reason why expert can't be placed next to a collector, link me to the discussion? o:) I was searching on both intelmq and manager trackers but with no luck.)

[ghost]

That would effectively solve use case of MaltaCIP. :) They have a file collector that receives all shadowserver files in a directory whose contents should be divided into shadowserver parsers.

Haven't heard of that issue yet. But we can solve that use case differently. Namely by providing the file name in the report as extra.file_name and then using the field in the parser to determine the type of feed. HTTP, RT and Mail collectors already save this kind of information in the report, I'll add it for the file collector now. For the required changes in the shadowserver parser I opened #1442

I advised to use a sieve bot that will distribute events to according parsers. But as an expert, sieve cannot be placed between collector and parsers.

(Could you please remind me of the reason why expert can't be placed next to a collector, link me to the discussion? o:) I was searching on both intelmq and manager trackers but with no luck.)

That limitation does only exists in the GUI, if you just configure it, it works fine. Maybe some experts require fields only existing in events, but that could always be the case.