search

framed-data / overseer

Overseer is a library for building and running data pipelines in Clojure.
Eclipse Public License 1.0
97 stars 10 forks source link

0.3.0: Allow specifying job handlers as maps #16

Closed andrewberls closed 9 years ago

andrewberls commented 9 years ago

Before, job handlers could only be functions accepting a job map. This extends the system such that maps can be used as job handlers, specifically of the form

  {:pre-process (fn [job] ...)
    :process (fn [job] ...)
    :post-process (fn [job res] ...)}

where res in post-process is the arbitrary value (if meaningful) returned from the main process job. Note that if exceptions will halt control flow; if either pre-process or process throw, no future steps will run!

elliot42 commented 9 years ago

:+1:

why were we doing this?

andrewberls commented 9 years ago

The ultimate goal here was to have jobs be more declarative in terms of their inputs/outputs. pre-processing is a nice place for us to pull in all our dependencies, and for example jobs can return a map/vector of generated parts and post-processors can upload those - there could even be a single shared util postprocessor that just uploads all artifacts