Closed L3o-pold closed 4 years ago
Hello @L3o-pold!
This error is not related to Sentinel. In fact, if we review the stack trace, it doesn't look like the application is configured to use Sentinel connections for Horizon because the ZADD
command executed through Laravel's standard Redis connection wrapper. If you'd like to use this package for Horizon, please read this section for configuration instructions.
Because the application isn't resolving Horizon's RedisQueue
implementation, it generates the string job ID shown in the debug message (HyLbvZ2kbjFlznzM0nHMrOuOF1Ap47zV
) instead of an integer. The error occurs because Horizon tries to execute the ZADD
command using the string ID for the "score" argument that Redis expects a number for.
When the application is configured to use this package's connections for Horizon (via the redis-sentinel
queue connection), it binds the appropriate RedisQueue
implementation needed to generate numeric job IDs for Horizon.
Hi @cyrossignol, thanks for your quick reply. That's what I was thinking but it should use redis-sentinel driver.
My .env
file contains:
HORIZON_DRIVER=redis-sentinel
CACHE_DRIVER=redis-sentinel
QUEUE_DRIVER=redis-sentinel
QUEUE_CONNECTION=redis-sentinel
REDIS_DRIVER=redis-sentinel
My config/horizon.php
contains
'driver' => env('HORIZON_DRIVER', 'redis'),
'use' => 'queue',
'waits' => [
'redis-sentinel:queue' => 60,
'redis-sentinel:redis-sentinel' => 60,
'redis-sentinel:default' => 60,
],
'environments' => [
'development' => [
'supervisor-1' => [
'connection' => 'redis-sentinel',
'queue' => ['high', 'medium', 'default', 'low', 'heavy'],
'balance' => 'auto',
'processes' => 96,
'min-processes' => 0,
'tries' => 1,
'timeout' => 3600,
'memory' => 32,
'max-process-for-queue' => [
'heavy' => 5,
]
],
],
],
My config/queue.php
contains:
'default' => env('QUEUE_CONNECTION', 'sync'),
'connections' => [
'redis-sentinel' => [
'driver' => 'redis-sentinel',
'connection' => 'queue',
'queue' => 'default',
'retry_after' => 3666,
'expire' => 3666,
],
],
My config/database.php
contains
'redis' => [
'client' => env('REDIS_CLIENT', 'predis'),
'queue' => [
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => env('REDIS_DB', 0),
'cluster' => 'redis',
],
Maybe I'm missing something in the configuration...
An important point is that the job is pushed from a other project (not with a Laravel Framework) but using some Illuminate package for the queue and the laravel-redis-sentinel package.
When pushing a job from the Laravel project (the one who's executing the queue), if the job failed, there is no Predis error.
But when the job is pushed from the other project, if the job failed, there is the Predis error.
Because it's not a Laravel project (so there is no horizon in it) it doesn't push information in the redis with the key horizon:job_id
or horizon:recent_jobs
. Maybe it came from here.
An important point is that the job is pushed from a other project (not with a Laravel Framework) but using some Illuminate package for the queue and the laravel-redis-sentinel package.
@L3o-pold, you're right--that is an important hint. When the job is pushed from the non-Laravel project (without Horizon), it sets the job ID to the string value shown above instead of to a number as Horizon expects it to be. The base RedisQueue
implementation in illuminate/queue
is not Horizon-aware, and this package cannot re-bind Horizon's implementation unless the package is available.
I think it's possible to install Horizon without Laravel, but I never tried this myself. You can also extend Laravel's RedisQueue
to override the createPayloadArray()
method to replace the the job ID generation logic with an approach that's compatible with Horizon. Then, extend Laravel's base queue RedisConnector
class to instantiate the custom RedisQueue
and reconfigure the QueueManager
of the non-Laravel application to use this class instead for Sentinel connections.
An easier approach could be to just register a callback for job serialization that replaces the job ID. You can try adding something like this to the bootstrap process:
use Illuminate\Queue\Queue;
...
Queue::createPayloadUsing(function ($connection, $queue, $payload) {
if ($connection === 'redis-sentinel') {
$payload['id'] = // generate horizon job ID...
}
return $payload;
});
...with the caveat that this hook system seems to be used for Laravel's test suite, so I'm not sure how well-supported the approach will be in the future.
I think we cannot use the createPayloadUsing
method for editing the id. It's generated later...
We extended the RedisQueue
and it works perfectly. Thanks for all those advice ❤️
@L3o-pold You're welcome! I'm glad it works.
Every time a job failed, we got an exception in the worker.
When we log the command sent to redis before the exception:
Is this related to redis sentinel?