Maintain ready-jobs as a set, not vector

framed-data / overseer

Overseer is a library for building and running data pipelines in Clojure.

Eclipse Public License 1.0

97 stars 10 forks source link

Maintain ready-jobs as a set, not vector #58

Closed andrewberls closed 8 years ago

andrewberls commented 8 years ago

This allows to use disj, which is much cleaner than a linear filter to accomplish the task of removing a job from the cache. Verified with local run

elliot42 commented 8 years ago

Does this still preserve shuffling? Or are all the nodes going to attempt to dequeue all the same jobs in the same order?

andrewberls commented 8 years ago

Both the new and old structure go through same unchanged lottery scheduling/shuffle

elliot42 commented 8 years ago

:+1: