Dramatiq operations on builtin brokers are thread-safe
however the broker in this repo is not built-in and I can't see any mention of thread safety nor any locking.
As far as I can see when a dramatiq task is enqueued then the following generic call chain occurs:
Actor.send > Actor.send_with_options > Broker.enqueue
Looking at SQSBroker.enqueue in this repo (which doesn't do any locking):
it does queue.send_message where queue comes from self.queues[queue_name]
the queue object originally comes from self.sqs.create_queue(...) where self.sqs is a boto3 "Resource"
Given the above, is this broker potentially not thread safe, e.g. for simultaneously enqueuing from multiple threads of a web server process (such as gunicorn) to the same queue?
Looking at your notes, I think you're probably right. I had missed that detail about boto resources. If anyone wants to fix this, I'd be happy to accept a PR.
I noticed in https://github.com/Bogdanp/dramatiq/blob/master/docs/source/troubleshooting.rst it says:
however the broker in this repo is not built-in and I can't see any mention of thread safety nor any locking.
As far as I can see when a dramatiq task is enqueued then the following generic call chain occurs:
Actor.send
>Actor.send_with_options
>Broker.enqueue
Looking at
SQSBroker.enqueue
in this repo (which doesn't do any locking):queue.send_message
wherequeue
comes fromself.queues[queue_name]
self.sqs.create_queue(...)
whereself.sqs
is a boto3 "Resource"sqs.create_queue
returns aQueue
resourceGiven the above, is this broker potentially not thread safe, e.g. for simultaneously enqueuing from multiple threads of a web server process (such as gunicorn) to the same queue?