search

ostark / craft-async-queue

Async Queue Handler for Craft 3 and 4
MIT License
93 stars 7 forks source link

Will this break with multiple Craft app instances? #46

Closed nickdunn closed 2 years ago

nickdunn commented 3 years ago

I've been using this plugin for a while now and I love it, thank you for developing it!

One of my Craft sites is hosted in Azure with autoscaling enabled i.e. Azure will spin up multiple instances of the web app with the same configuration, pointing to the same db and Redis cluster. I am not currently using Redis for the queue, and I still have the queue running automatically (all Craft defaults). However I am running craft queue/run on a 5 minute cron to catch any instances where the queue might be jammed.

First question: is this cron necessary? Or is this only needed if runQueueAutomatically is disabled? Can I safely remove it? I do sometimes see the queue stopped, with items sitting in it, and the only way I can manually flush is to run the Async Queue Test which gets it going again.

Second: what will happen when my app is scaled? Obviously this means more worker processes since the queue is running in each instance. Does Craft and this plugin do locking to make sure items on the queue aren't grabbed by more than one worked? Are these locks based on the file system (and therefore only work in a single instance) or at the db/Redis level?

Many thanks!

ostark commented 2 years ago

Sorry for the late reply.

In my experience, running multiple concurrent queue workers (craft queue/run) with Craft's database queue CAN lead to deadlocks.

I'd say, you can safely use Async Queue on multiple HTTP frontends as long as you set ASYNC_QUEUE_CONCURRENCY to 1.

First question: is this cron necessary?

Only if someone schedules a job with a delay

Second: what will happen when my app is scaled?

In theory, it does not matter how many instances you use. The "locking" does not rely on the local file system, but on the database, which means you can not use Redis.

If you expect a serious amount of queue jobs, better use a supervisored queue runner and Redis.

nickdunn commented 2 years ago

Great, thanks for the feedback! I have since changed ASYNC_QUEUE_CONCURRENCY to 1 which I think has reduced deadlocks (but not fully) and we seem to be running two app instances OK.

I do find that sometimes the queue is stuck and the only way seems to be running the Async Queue Test utility that kicks it back into processing again. I'm not sure why this happens and it's why I was thinking of running queue/run periodically via a cron. Have you encountered this before?

ostark commented 2 years ago

I'm not a "user", but from time to time users experience stucking queue messages. That's probably not an issue with this plugin, but with Craft/Yii, as the plugin simply runs craft queue/run.