Best way how to run Laravel queue worker & cron jobs in Swarm?

[AdrianSkierniewski]

Hi Bret,

First of all, I'd like to thank you for a very good course on Udemy "Docker Swarm Mastery", I'm looking forward to watch new lessons.

Some time ago (early days for Docker) I created a little bit different setup for our Laravel app. Since then I was just tweaking it, but now I'm trying to solve some fundamental issues with it. There are couple similarities. For example, I used a single container to run nginx and php-fpm with supervisord. But there are some differences too - developers don't need to build app image, they're just downloading it from docker hub and the whole build process is later a part of a GitLab CI/CD pipeline.

My production setup still uses Ansible & docker-compose and I'd like to start using Docker Swarm. The main problem with my current approach as is that I'm, not only running nginx & php-fpm in my container but there is a Cron & Laravel Queue Worker there. Evertything is managed by Supervisord proces.

This works on single VPS with docker-compose but it won't scale in Docker Swarm.

I'd like to start deploying in to Swarm and to do that I'd need to solve couple problems: 1) Laravel Queue Worker (aka Horizon - it's more robust version of it) uses Redis, I'd need to gracefully shutdown it during deployment using php artisan horizon:terminate and I don't want to fight with Supervisord trying to restart it, so it would be nice to have it in separate container. 2) If I'd like to split queue worker from app (php-cli vs php-fpm + nginx) I'd need to support two images and how should I share same code base between those two images inside Swarm? Probably I'd need to build two images with same code base as part of my GitLab pipelines. Right now I don't see any other alternative. 3) Next I could start scaling web_workers & queue_workers separetly and change them one by one during deployment but I'm not sure how I could send some signal (docker exec would be nice but it won't work in Swarm) for queue_workers to gracefully shutdown before deployment. I need to make sure that I wont interupt any jobs. 4) Next problem would be related to Cron, if I'd leave it as is, it will duplicate cron runs by number of web_workers, so I'd probably need another image just to run cron jobs. I don't see any other solution right now. 5) Last problem is strictly related to gathering all those logs for all those apps. Right now I'm using some script to run tail on fifo. I'd probably need to leave this as is.

Everything starts looking as a realy big app but I still want to put that on single server by default and scale when it's needed.

I'd like to ask you on your thoughts on this problems? Have you encountered them? How people are solving this kind of issues. Any feedback will be very helpfull to me.

[BretFisher]

Whew there's a lot there. Let me see if I can give you some guidance without a novel (which I sometimes do 🤓)

• Single image for php/nginx: This is one of the very few exceptions I have for using supervisord and let them both run in an image across a file socket. The argument I have is they are useless without each other, and a network between them would only add latency. If you scaled one, you'd scale the other most likely. The way they work together doesn't make nginx a typical reverse proxy per se. If you were using nginx as a separate HTTP proxy, I'd say it should be its own image. Note that just because these two are in the same image, doesn't mean I recommend putting more services in that image to run at the same time.
• Workers and scheduled jobs: What I do recommend is having a single image with all the things you need to do those jobs as their own swarm services, and change the command: at runtime to do something other then start supervisord. I'm no Laravel expert but I think you could then do scheduled things in code, one service for each type. If you needed to scale a worker job up, you could ramp up the replicas of that service independently. For example, to run a worker you could just use this in a compose file with the same image as nginx/php:

command: ["php", "/var/www/app/artisan", "queue:work", "--daemon", "--queue=do-work", "--sleep=3", "--tries=3"]

• For task shutdown during a update, Docker will give the running app in a container a STOPSIGNAL, which you can adjust, and your code should listen for that signal and do proper cleanup. Docker will wait (by default 10s). This is a distributed computing necessity... that is, using Linux shutdown signals to properly exit apps. Do require some other thing manually run or externally run isn't a best practice. I'm not sure if someone has solved this for your use case, but it's a 12 Factor guideline that "disposability" be built into the running app.
• All your logs should be mapped to stdout/stderr in the Dockerfile so they are all dumped to Docker. The official nginx does this, and it's the way all logs should be redirected if they can't be sent to stdout/stderr by default. Then use docker log drivers in your services to centralize those logs in a long-term system like ELK or Papertrail, etc.

[AdrianSkierniewski]

Hi,

Thank you for a detailed answer. Since I wrote this ticket I manage to solve all my problems but I took slightly different approach. I've created two images instead one, first for nginx+php and second very similar only with php-cli. During my work on those two images, I learn a lot of new things about docker & docker swarm. I've handled all UNIX signals in my containers. Fixed some permissions & UNIX signals issues by changing to gosu and a lot more. Here is a repository with Docker files for those two images: https://github.com/GrupaZero/platform-container/tree/master/v5

With that new knowledge and after reading your response now I see that I could use ENTRYPOINT and then CMD to run other commands in the container. I'll probably try to do that later but for now, I'll just stay with those two images.

Besides that, I've created stack file to deploy the whole app on a new server just after provisioning it with Ansible. I'm doing this only at the beginning because I'd like to have a full control of the whole rolling deployment process and stacks doesn't give me that control. I had even more problems to solve during deployment, for example, I need to create some backups & run Laravel database migration in a swarm. Bellow, I've attached links to some ansible playbooks if someone wants to see how I solve these problems.

The whole deployment & rollback process isn't perfect and it is a lot more complicated because there could be a lot of edge cases when you need to decide what you can rollback and how to do it. I'll probably write a custom tool just to run it through ansible and talk with docker API and decide when deployment was successful.

Stack file: https://github.com/GrupaZero/platform/blob/master/ansible/deploy-stack.yml Deployment process: https://github.com/GrupaZero/platform/blob/master/ansible/deploy-app.yml

[ohnotnow]

If anyone else comes across this - I based my docker laravel queue/schedular off of the ideas in this Laravel News article : https://laravel-news.com/laravel-scheduler-queue-docker . Also possibly of interest, they have an article on using multi-stage builds with a Laravel project : https://laravel-news.com/multi-stage-docker-builds-for-laravel

Anyway - I'm using a those ideas tweaked a bit here and there and it's working out fairly well. So far... ;-)

[a-h-abid]

If anyone else comes across this - I based my docker laravel queue/schedular off of the ideas in this Laravel News article : https://laravel-news.com/laravel-scheduler-queue-docker .

Hi @ohnotnow , I'm also following this article to run my app, queue & scheduler. All running fine, just have issue with graceful shutdown for queue & scheduler. Have you found any way for them? When try to stop, they always wait for 10 sec & then gets force kill.

[iraklisg]

Hi Bret, All

I am using separate services for nginx, php-fpm and db in production.

I also use a custom Dockerfile for my application that extends from official php-fpm, copies the required code/dependencies and also uses an entrypoint that, among other tasks, check if the db connection is present and runs any migrations that have to be run

# Dockerfile

FROM php:8-fpm
...
COPY --chown=www-data ./ /var/www/html/
...
ENTRYPOINT ["app-entrypoint.sh"]
CMD ["php-fpm"]

I try to run a queue worker directly from my app service by adding this line into my entrypoint script

# entrypoint-app.sh

#!/bin/bash

set -e
...
php artisan migrate --force

php artisan queue:work --queue=default --sleep=1 --timeout=90 --tries=5  # <-------- this line here

The problem is that I cannot make artisan queue:work to run in the background and as a result, the Dockerfile CMD ("php-fpm") is not running and the website hangs.

Is there a way to have worker started without using a dedicated service for this? The reason that I do not want to use a separete service is that php artisan queue:work needs to have access to my codebase and thus, if I use a dedicated "worker" service I have to copy the code into this service's image. That's why I want to use the app service that includes the code (in production) and have the queue worker to start along with php-fpm

[BretFisher]

I would follow the same advice I posted earlier in this thread:

Use the same single image and don't use entrypoint to start a separate long running process, that's an anti-pattern and likely won't work. Rather, in the second service use the same image and change the command in your YAML