Transformation to support whether a pipeline is applied or not

[joshnck]

Right now, the way pipelines work is they are first prioritized and then applied in priority order. Every pipeline is applied to every rule, and it is up to the author of the pipeline to come up with complex logic to assign transformations based on the Sigma rule at hand.

My proposal is to create the opportunity to filter out which pipelines are applied based on conditional logic in a priority 1 pipeline object.

For example, imagine you've got 3 pipelines:

• windows.yml
• linux.yml
• mac.yml

In those pipelines you'll do a basic transformation like this, but swapping windows for linux or mac:

name: windows
priority: 20
transformations:

  - id: win_index
    type: add_condition
    conditions:
      index:
        - "windows"
    rule_conditions:
      - type: logsource
        product: windows

In the default state every pipeline is handled, and it is up to the rule conditions to state whether a specific transformation object is applied.

What I propose is to include a transformation type that can be used in a priority: 1 pipeline that says something like:

name: pipeline_assignment
priority: 1
transformations:

  - id: default_to_false
    type: pipeline_state
    name:
      - windows
      - linux
      - mac
    applied: false

  - id: win_rule
    type: pipeline_state
    name: windows
    applied: true
    rule_conditions:
      - type: logsource
        product: windows
  - id: win_rule
    type: pipeline_state
    name: linux
    applied: true
    rule_conditions:
      - type: logsource
        product: linux
  - id: win_rule
    type: pipeline_state
    name: mac
    applied: true
    rule_conditions:
      - type: logsource
        product: mac

Then this pipeline_state transformation object will tell the upstream converter that only objects with the applied: true state will apply.

In order to avoid breaking everyone's pipelines, the default state for applied: should be true and that is why I am using a generic default_to_false to override that state to false with a list of pipeline objects (called by name) then I am specifically stating which pipelines I'd like supported based on conditional logic.

[thomaspatzke]

There's already a set_state transformation that could be used for this purpose. I also have a plan to add nested processing pipelines as additional tranformation, which would basically allow such constructs:

transformations:
  - id: some_block
    rule_conditions:
      - type: processing_state
        key: pipeline_something
        val: true
    type: pipeline
    pipeline:
      transformations:
        - id: field_mappings
          ...
        - id: field_name_prefix
          ...

This would add much flexibility to processing pipelines and also cover this use case.

[thomaspatzke]

Just pushed nested pipelines to the main branch and release 0.11.12 will contain this.

[thomaspatzke]

Doc with YAML example