Task queue for Aleph and ingest-file backed by RabbitMQ

alephdata / servicelayer

Common interface definitions for aleph toolkit services and applications

MIT License

7 stars 5 forks source link

Task queue for Aleph and ingest-file backed by RabbitMQ #63

Closed sunu closed 2 years ago

sunu commented 2 years ago

Part of fixing https://github.com/alephdata/aleph/issues/2127

Implements a worker that listens on one or more RabbitMQ queues.
Queue state per collection is stored in Redis and it tracks finished task count as an integer and pending tasks and running tasks as a set
Late acknowledgement is enabled in RabbitMQ. So tasks are retried if a worker disconnects without sending an acknowledgement

sunu commented 2 years ago

@Rosencrantz I want to merge this and make a minor release. Since we have added the new worker implementation in a different module, merging this branch won't break anything that use the old worker implementation. What do you think?

Rosencrantz commented 2 years ago

@Rosencrantz I want to merge this and make a minor release. Since we have added the new worker implementation in a different module, merging this branch won't break anything that use the old worker implementation. What do you think?

To be clear, you're looking to do some sort of beta or RC type release as discussed right?

Rosencrantz commented 2 years ago

This looks fine, the work here (in the most part) seems additive rather than changing existing code so I'm not too concerned about it having a detrimental impact. Also this is to be used as a 'beta' for further testing.