framed-data / overseer

Overseer is a library for building and running data pipelines in Clojure.
Eclipse Public License 1.0
97 stars 10 forks source link

Add api/validate-graph-handlers #22

Closed andrewberls closed 9 years ago

andrewberls commented 9 years ago

This is an optional function which accepts a set of handlers and a job graph, and asserts that all handlers referenced in the graph exist.

laurakelly commented 9 years ago

:+1:

elliot42 commented 9 years ago

So, this looks good in general, but originally we intentionally let workers be able to only handle subsets of the jobs in the queue. This was because leaving that option open allowed us to have multiple configurations or classes of workers, each of which could be tuned to handling certain subsets of jobs. This strategy disallows that option, which is good in some senses but does limit our options (e.g. we cannot run a worker pool with normal memory for most jobs but high amounts of RAM for ML jobs)