Migrate to Quorum queues in rabbitmq and setup message poisoning & deadletter queues

lfse-slafleur commented 5 months ago

Ask Serkan (TNO) about his work
Maybe Celery can use quorum queues? If so, perhaps that would make the event queue also durable?

lfse-slafleur commented 3 months ago

Did some investigating:

Classic queues do not support 'Poison message handling' (https://www.rabbitmq.com/docs/quorum-queues#poison-message-handling). This prevents us from using a delivery-limit with classic queues and requires us to move to quorum queues.
Quorum queues do not support priorities so we will be giving up support from RabbitMQ with this move (https://www.rabbitmq.com/docs/quorum-queues#feature-matrix). While message priorities may become important later, currently no support for prioritisation is required.
Celery does not support quorum queue's out of the box but official support is incoming: https://github.com/celery/celery/issues/6067 It is possible to create the tasks queues as quorum queues by adding an extra argument when defining the queue: https://docs.celeryq.dev/en/latest/userguide/routing.html#rabbitmq-message-priorities . However, other features such as ETA tasks may not work. Nevertheless, we do not use ETA tasks or any of the other features discussed but we would need to confirm if we have any issues with moving to quorum queues by trying it out.

lfse-slafleur commented 3 months ago

Decided that delivery-limit functionality is required currently to solve the issue of having a job be retrieved definitely when it causes an OOM.

Prioritization is still important as well because we need a mechanism to combat having a single user fill up the queues with large amounts of jobs and not being able to have another user's job overtake the existing jobs.

Nusnus commented 2 months ago

Celery does not support quorum queue's out of the box but official support is incoming: support for quorum queues celery/celery#6067

On it’s way: https://github.com/celery/celery/pull/9121

Nusnus commented 2 months ago

Also, per the topic of this issue, see: https://www.rabbitmq.com/docs/migrate-mcq-to-qq

lfse-slafleur commented 2 months ago

Celery does not support quorum queue's out of the box but official support is incoming: support for quorum queues celery/celery#6067

On it’s way: celery/celery#9121

@Nusnus Thank you Tomer for the references and for adding this to Celery, much appreciated!

lfse-slafleur commented 2 months ago

Support for Quorum queues is available in Celery main release available soon. Switching over to component tickets for tracking and closing this one.

Nusnus commented 2 months ago

Support for Quorum queues is available in Celery main release available soon. Switching over to component tickets for tracking and closing this one.

Celery v5.5.0b1 was released with the new feature support.

@lfse-slafleur

lfse-slafleur commented 2 months ago

Support for Quorum queues is available in Celery main release available soon. Switching over to component tickets for tracking and closing this one.

Celery v5.5.0b1 was released with the new feature support.

@lfse-slafleur

@Nusnus Thanks again Tomer! Great work!

Project-OMOTES / architecture-documentation

Migrate to Quorum queues in rabbitmq and setup message poisoning & deadletter queues #25