onyx-platform / onyx

Distributed, masterless, high performance, fault tolerant data processing
http://www.onyxplatform.org
Eclipse Public License 1.0
2.05k stars 205 forks source link

Flow conditions improvements #816

Closed lbradstreet closed 7 years ago

lbradstreet commented 7 years ago

Issue 1:
 Short circuit exception flow condition such as:

{:flow/from :input-stream
 :flow/to [:error-task]
 :flow/short-circuit? true
 :flow/thrown-exception? true
 :flow/post-transform :my.ns/post-transform
 :flow/predicate :my.ns/handle-error?}

Should: Only flow to :error-task for exceptions that pass that predicate. Flow to all task other than :error-task the rest of the time?

Breaking changes in this solution: All regular segments currently flow to :error-task.

Issue 2: Flow conditions, windowing, trigger/emit:

Suggest we allow:

{:flow/from [:my-task :my-trigger]
;; or :flow/from-trigger :my-trigger
 :flow/to [:task1]}

When used, all other flows will be cut off from :my-task, even if :my-task is not named? 
Breaking change here: All all trigger/emit flows will be cut off if someone is currently using a flow condition + trigger/emit.

Issue 3: Triggered segments should go through to output plugin on output tasks with windows. Everything will look the same from users point of view, implementation tbd.

lbradstreet commented 7 years ago

Better discussed in #819.