arobson / rabbot

Deprecated: Please see https://github.com/Foo-Foo-MQ/foo-foo-mq
MIT License
277 stars 129 forks source link

How to consume from primary and secondary job queue #190

Open EwaRvr opened 5 years ago

EwaRvr commented 5 years ago

Is there a way to consume from 2 queues in such a way that as long as there are messages in the primary queue, messages in the secondary queue are ignored?

Scenario: Our API posts messages into a queue, and waits for a certain time for a response. If that response comes back within a certain time, it return that response, but if it times out due to the backend being flooded with requests, it sends back a 201 Accepted status. In addition, we want to take that request, and place it in a separate (lower priority) queue, probably using deadlettering. The reason is that as now there is no API client waiting for the response anymore, we might as well prioritize other requests that a client is waiting for, over this one, and come back to this one once the spike has quieted down.

But now our challenge is how to have a single process consume from our primary job queue first, and only consume from the secondary/lower priority queue when there is nothing left in the primary queue?

I know RabbitMQ has message priority within a single queue, which Rabbot doesn't support, but I'd much prefer using separate queues.

Using Rabbot, what is the best strategy for this?

Thanks

woolite64 commented 5 years ago

Rabbot allows you to set maxPriority on queue creation. See maxPriority in the Topology documentation.

EwaRvr commented 5 years ago

@woolite64 Yes, but as arobson replied to issue #13, it does not support setting priority on the messages. meaning they ALL go at priority level 0.

alonisser commented 4 years ago

@EwaRvr while I would the pr to be merged, we currently maintain a fork https://github.com/Zappatta/rabbot by the talented @Zappatta allowing message priorities.. and we do use it successfully