arobson / rabbot

Deprecated: Please see https://github.com/Foo-Foo-MQ/foo-foo-mq
MIT License
277 stars 129 forks source link

Need help designing a specific use case: handle asynchronous tasks on the same resource across consumers #176

Open booboothefool opened 6 years ago

booboothefool commented 6 years ago
                  |--> queue_for_resource1 - consumer1
exchange          |--> queue_for_resource2 - consumer2
                  |--> queue_for_resource3 - consumer3
                  |--> ...

queue_for_resource# needs to be dynamically generated where resource# is some unique id. Because of this, I can't just put it all in the topology config because I don't know how many or which ones I will need ahead of time. I assume should be dynamically generated because:

Only 1 consumer should be able to process from a queue for a resource at a time. This is because the tasks in the queue will eventually do calls that modify that particular resource e.g. in a database, so if 2 consumers try to process at the same time, the resource will lock and fail. I was looking at prefetch: 1, exclusive queue, and exclusive consumer, and while I'm not sure of the differences, I believe exclusive consumer is the solution?

I think the solution is a combination of:

Another concern is what happens if consumer1 is assigned to queue_for_resource1, but consumer1 goes down? Will another consumer, such as consumer2 automatically subscribe and pick up where it left off?