warpstreamlabs / bento

Fancy stream processing made operationally mundane. This repository is a fork of the original project before the license was changed.
https://warpstreamlabs.github.io/bento/
Other
1.06k stars 71 forks source link

add: Strict Mode option to Pipeline configuration #174

Open gregfurman opened 2 days ago

gregfurman commented 2 days ago

Motivation

We need a strict_mode to allow for a processing step to return and log an error, with the subsequent processors not being run.

Related to https://github.com/warpstreamlabs/bento/pull/166 -- where we opted for nacking at the failed processor instead of at the output layer.

Can be used as follows:

pipeline:
  strict_mode: true
  processors:
    - mapping: |
        root = throw(this.value)
    - mapping: |
        root.message = "I'm never reached..."

Changes

TODO