Configuring the queue to be processed by a separate server to the main application server adds the job to the wrong redis store

matt-ostrom-hall-rss commented 2 years ago

Horizon Version: 5.7.17
Laravel Version: 8.76.2
PHP Version: 7.4.24
Redis Driver & Version: predis 1.1.9
Database Driver & Version: MySQL 5.7.36

Description:

If you configure a custom redis connection for an external server in config/database.php and then set that as the redis queue connection in config/queue.php, queued jobs are added to the local redis store on the application server instead of being pushed to the external server redis store that is configured for use.

Steps To Reproduce:

Install a fresh laravel application and horizon.

Add a new entry to the redis array in config/database.php and add the environment variables for the external redis server

'queue' => [
'url' => env('REDIS_QUEUE_URL'),
'host' => env('REDIS_QUEUE_HOST', '127.0.0.1'),
'password' => env('REDIS_QUEUE_PASSWORD', null),
'port' => env('REDIS_QUEUE_PORT', '6379'),
'database' => env('REDIS_QUEUE_DB', '0'),
]

Update the connection value to 'queue' for the redis entry in config/queue.php

'redis' => [
'driver' => 'redis',
'connection' => 'queue',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
'after_commit' => false,
]

Create a job and dispatch it
The job will be a pending job on the server the job was dispatched from and not be in the redis store on the external server.
https://github.com/Really-Simple-Systems/horizon-bug-report

driesvints commented 2 years ago

Are you sure all your env variables are set correctly? Because the Redis connections are identical. Is REDIS_QUEUE_HOST pointing to a different url than REDIS_HOST?

matt-ostrom-hall-rss commented 2 years ago

Yeah the REDIS_QUEUE_HOST variable is pointing to the external redis server, while REDIS_HOST is pointing to the application server.

driesvints commented 2 years ago

I have to be frank: this sounds like a configuration issue as I know of specific apps that are using this properly. I'm fairly sure there's something incorrectly configured on your end.

matt-ostrom-hall-rss commented 2 years ago

This happens with the fresh repo I've linked above, the only configuration that's been done is changing the queue connection as above, without horizon installed the job gets put on the redis store on the external server. Any ideas on what could be configured incorrectly? It works if I leave the queue connection as the default redis connection, however we can't do this and change the default connection host, as we want the sessions to remain on the application server.

driesvints commented 2 years ago

I wonder if it has to do with these settings for the supervisor: https://github.com/Really-Simple-Systems/horizon-bug-report/blob/main/config/horizon.php#L169-L170

Also: what's the value of QUEUE_CONNECTION for you?

matt-ostrom-hall-rss commented 2 years ago

I've got it set as QUEUE_CONNECTION=redis

driesvints commented 2 years ago

I'm out of options I'm afraid. I don't know why this is happening. Maybe one of my team mates knows but most of us are with vacation until the beginning of the year. I'll leave this open for a bit longer but I'm afraid I can't help you further.

matt-ostrom-hall-rss commented 2 years ago

Ok thank you for your help, i'll check back in the new year. Have a good holidays :)

driesvints commented 2 years ago

@jmarcher I'm gonna have a more thorough look at this soon hopefully. In the meantime can you maybe prepare a repo for me to try out with:

laravel new bug-report --github="--public"

Please make sure that you have the latest version of the Laravel installer in order to run this command. Please also make sure you have both Git & the GitHub CLI tool properly set up. Please do not amend and create a separate commit with your custom changes. After you've posted the repository, we'll try to reproduce the issue.

And also list the ENV variables that I should set and prep them a bit some values you're using. I'll then try to recreate the problem myself.

Thanks!

mattostromhall commented 2 years ago

@driesvints was this comment intended for this issue? I've created a repo to help reproduce here https://github.com/Really-Simple-Systems/horizon-bug-report

driesvints commented 2 years ago

Ah yeah, sorry my bad. Can you also prep an .env file for me to use?

matt-ostrom-hall-rss commented 2 years ago

Yeah sure, only bits you need to set specifically for you is DB_DATABASE and REDIS_QUEUE_HOST I think. Thank you :)

APP_NAME="Horizon Test"
APP_ENV=local
APP_KEY=base64:0RcS4lKkogxrZSL/flqlDO6TXsKwzfRUjE7i/wfyCvU=
APP_DEBUG=true
APP_URL=http://HorizonTest.test

LOG_CHANNEL=stack
LOG_DEPRECATIONS_CHANNEL=null
LOG_LEVEL=debug

DB_CONNECTION=sqlite
DB_DATABASE=/your-path/HorizonTest/database/database.sqlite
DB_USERNAME=root
DB_PASSWORD=

BROADCAST_DRIVER=log
CACHE_DRIVER=file
FILESYSTEM_DRIVER=local
QUEUE_CONNECTION=redis
SESSION_DRIVER=file
SESSION_LIFETIME=120

MEMCACHED_HOST=127.0.0.1

REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379

REDIS_QUEUE_HOST=external-redis-ip-here
REDIS_QUEUE_PASSWORD=null
REDIS_QUEUE_PORT=6379

MAIL_MAILER=smtp
MAIL_HOST=mailhog
MAIL_PORT=1025
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_ENCRYPTION=null
MAIL_FROM_ADDRESS=null
MAIL_FROM_NAME="${APP_NAME}"

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=us-east-1
AWS_BUCKET=
AWS_USE_PATH_STYLE_ENDPOINT=false

PUSHER_APP_ID=
PUSHER_APP_KEY=
PUSHER_APP_SECRET=
PUSHER_APP_CLUSTER=mt1

MIX_PUSHER_APP_KEY="${PUSHER_APP_KEY}"
MIX_PUSHER_APP_CLUSTER="${PUSHER_APP_CLUSTER}"

driesvints commented 2 years ago

Hey @matt-ostrom-hall-rss, sorry it took me a while to debug this. I cannot reproduce this. For me, the job is stored in a separate database:

I didn't exactly used a different host but I did had to set REDIS_QUEUE_DB to REDIS_QUEUE_DB="2" to make sure it was stored in a separate database. Maybe you forgot to set this value? By default it points to 0 in your config which is the same database as the one in your default Redis connection. I realize this can be 0 because you're using different hosts.

I'm sorry I couldn't exactly try this with real different hosts. I think it's best if you're still experiencing issues here to try a support channel.

sp4r74cus commented 2 years ago

Hi @driesvints

I faced the same issue today when trying to separate queues to different connections.

Horizon Version: 5.9.6 Laravel Version: 9.8.1 PHP Version: 8.1.4 Redis Driver: phpredis 5.3.5 Database Driver & Version: MySQL 8

Steps To Reproduce:

created a clean Laravel app and installed Horizon.

configured new redis connection (Digital Ocean redis database) in config/database.php

'redis' => [
 ...

  'digitalocean' => [
        'url' => env('DO_REDIS_URL'),
    ],
]

configured a new queue connection that uses the DO server in config/queue.php

    'digitalocean-redis' => [
        'driver' => 'redis',
        'connection' => 'digitalocean',
        'queue' => env('REDIS_QUEUE', 'default'),
        'retry_after' => 90,
        'block_for' => null,
        'after_commit' => false,
    ],

configured environments in config/horizon.php

    'digitalocean' => [
        'connection' => 'digitalocean-redis',
        'queue' => ['digitalocean'],
    ]

created 2 jobs, one dispatched on 'default' queue which is configured to process on the default redis connection (local) , and the second one dispatched on 'digitalocean' queue that is set to process on the digitalocean connection

.env

QUEUE_CONNECTION=redis

REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379

#REDIS_URL=rediss://default:*******@db-redis-mailcoach-do-user-353183-0.b.db.ondigitalocean.com:25061
DO_REDIS_URL=rediss://default:*******@db-redis-mailcoach-do-user-353183-0.b.db.ondigitalocean.com:25061

Results:

'default' queue is processed and 'digitalocean' queue job stays pending.

just to be sure all is set correctly, i set the default redis connection to digitalocean and all jobs dispatched and processed. so i still had 2 connections, and the queues were using both, and all jobs processed as expected.

then i switched connections:

'default' redis connection was configured to digitalocean redis server
'digitalocean' was now configured to the local redis server

results were now opposite. the job that dispatched on the remote server was processed and the other one one pending.

Hope that helps.

driesvints commented 2 years ago

@matt-ostrom-hall-rss I'm not sure if I'll ever be able to reproduce that. It worked for me the last time I tried. I'm not convinced there's an issue but rather a configuration issue. It's best that you try a support channel:

sp4r74cus commented 2 years ago

So it really was a configuration issue.

The jobs were dispatched without inConnection() value so they all went into the default one.

My confusion was that I thought Horizon handles that when I set the connection in the supervisor. But I now understand it only look for dispatched jobs there.

We learn a new thing every day :)

Thanks for your time replying and linking to those resources 🙏

laravel / horizon

Configuring the queue to be processed by a separate server to the main application server adds the job to the wrong redis store #1106

Description:

Steps To Reproduce: