seung-lab / python-task-queue

Asynchronous serverless task queue with timed leasing of tasks. Threaded implementations for SQS and local filesystem.
BSD 3-Clause "New" or "Revised" License
36 stars 9 forks source link

Task Dependencies #35

Open william-silversmith opened 3 years ago

william-silversmith commented 3 years ago

It would be useful to support task chaining. This allows processors to guarantee locality when acquiring the next task.

The simplest version of this would be to insert a list of tasks. A more sophisticated version would allow for the specification of a tree that would process breadth-first.

There are drawbacks to using this kind of processing mode. One of the big ones is that tasks with dependencies won't benefit from automatic parallelism and retries of individual tasks. Instead, the whole unit must complete or be retried.