search

seung-lab / python-task-queue

Asynchronous serverless task queue with timed leasing of tasks. Threaded implementations for SQS and local filesystem.
BSD 3-Clause "New" or "Revised" License
36 stars 9 forks source link

Queue Transfer #28

Closed william-silversmith closed 3 years ago

william-silversmith commented 4 years ago

It would be cool to have a way to copy or move a queue from one location / protocol to another.

For example, maybe you are experimenting with fq because it's easier to manage and then want to transfer the results to SQS directly instead of rerunning because the process took forever to compute the right tasks. This is the case for me right now with computing tasks from all the billions labels in a dataset.

william-silversmith commented 4 years ago

This is made possible by tq.tasks() in 3eeedc284b6d4674ca1c1c3221de0aed752a17e9 

tq_sqs = TaskQueue('sqs://...')
tq_fq = TaskQueue('fq://...')

tq_sqs.insert(tq_fq.tasks())
tq_sqs.insert(tq_fq)