daison12006013 / laravel-horizon-cluster

A laravel horizon with redis cluster support
MIT License
7 stars 3 forks source link

[Question] are the braces on queue names also required when dispatching the job to a specific queue? #4

Open shopapps opened 3 months ago

shopapps commented 3 months ago

I noticed in your config example you wrap the queue names with braces:

...
                'queue'      => [
                    '{redis-high}',
                    '{redis}',
                    '{redis-low}',
                ],
...

and googling tells me, this is one of the workarounds that people use to stop the CROSSSLOT errors. But my question is in my code to dispatch to one of these queues to i just use 'redis-high' or '{redis-high}' :-)

also if i was to set env HORIZON_PREFIX to be something like '{my_app_horizon}' (with braces) would this negate the need to use the queue name braces? meaning does their just need to be one set of braced string in the redis key to stop the error or can you have multiple?

The reason i ask is i have everything setup and i throw a load of jobs in via

Bus::batch($batches)->onQueue('redis-high')->dispatch();

and the jobs can be viewed in horizon under pending jobs, but are just not showing up in the Current Workload panel and are just not being processed. :-(

daison12006013 commented 3 months ago

@shopapps

I'm not so expert in terms of the whole abstraction of laravel horizon in terms of the prefix, but for sure the guys or the main laravel/horizon can answer your question about this.

But curly braces can be read here

and the jobs can be viewed in horizon under pending jobs, but are just not showing up in the Current Workload panel and are just not being processed. :-(

may I know what's your supervisord command list?

shopapps commented 3 months ago

Thanks for the reply. In my systemd/system folder i have a file with the following:

[Unit]
Description=Laravel Horizon Queue Manager
After=syslog.target
After=network.target

[Service]
Type=simple
User=www-data
Group=adm
WorkingDirectory=/srv/app/production/current/
StandardOutput=append:/srv/app/production/current/storage/logs/horizon.log
StandardError=append:/srv/app/production/current/storage/logs/horizon_errors.log
ExecStart=/usr/bin/php /srv/app/production/current/artisan horizon
ExecStop=/usr/bin/php /srv/app/production/current/artisan horizon:terminate
#PIDFile=/run/horizon.pid
Restart=always
RestartSec=5

# systemd stack hardening
ProtectSystem=full
PrivateDevices=yes
PrivateTmp=yes
NoNewPrivileges=true
ReadWritePaths=/var/log

[Install]
WantedBy=multi-user.target

i build a separate dedicated redis-server for now to get it working, but can go back to test at some point if you need it. I'm wondering if it has something to do with batches (which is what i was trying to use). I was batching a small number of jobs, which in turn hyrdated additional jobs into the batches. It worked locally on my dev (macos) setup but not in AWS.

regards,

Paul

daison12006013 commented 3 months ago

@shopapps the only thing I could advice is to do a trial and error.

  1. Have you tried to run php artisan horizon manually in your cli and see if it dispatches properly (do it in your own cloud provider / EC2 / Droplet or whatever)
  2. Have you tried supervisord instead of systemd?
  3. Have you tried to use onConnection('redis-high') and setup your config/queue.php in your laravel?
    • Maybe experiment and try to use curly-brace for your queues and see if the jobs getting processed. image