Exception from Predis when a job failed

[L3o-pold]

Every time a job failed, we got an exception in the worker.

[2019-09-12 08:50:10] development.ERROR: ERR value is not a valid float {"exception":"[object] (Predis\\Response\\ServerException(code: 0): ERR value is not a valid float at /var/www/html/vendor/predis/predis/src/Pipeline/Pipeline.php:102)



[stacktrace]

#0 /var/www/html/vendor/predis/predis/src/Pipeline/Pipeline.php(147): Predis\\Pipeline\\Pipeline->exception(Object(Predis\\Connection\\Aggregate\\SentinelReplication), Object(Predis\\Response\\Error))

#1 /var/www/html/vendor/predis/predis/src/Pipeline/Pipeline.php(166): Predis\\Pipeline\\Pipeline->executePipeline(Object(Predis\\Connection\\Aggregate\\SentinelReplication), Object(SplQueue))

#2 /var/www/html/vendor/predis/predis/src/Pipeline/Pipeline.php(215): Predis\\Pipeline\\Pipeline->flushPipeline()

#3 /var/www/html/vendor/predis/predis/src/Client.php(445): Predis\\Pipeline\\Pipeline->execute(Object(Closure))

#4 /var/www/html/vendor/predis/predis/src/Client.php(396): Predis\\Client->createPipeline(NULL, Object(Closure))

#5 /var/www/html/vendor/predis/predis/src/Client.php(418): Predis\\Client->sharedContextFactory('createPipeline', Array)

#6 /var/www/html/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(114): Predis\\Client->pipeline(Object(Closure))

#7 /var/www/html/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(214): Illuminate\\Redis\\Connections\\Connection->command('pipeline', Array)

#8 /var/www/html/vendor/laravel/horizon/src/Repositories/RedisTagRepository.php(103): Illuminate\\Redis\\Connections\\Connection->__call('pipeline', Array)

#9 /var/www/html/vendor/laravel/horizon/src/Listeners/StoreTagsForFailedJob.php(41): Laravel\\Horizon\\Repositories\\RedisTagRepository->addTemporary(2880, 'HyLbvZ2kbjFlznz...', Array)

#10 [internal function]: Laravel\\Horizon\\Listeners\\StoreTagsForFailedJob->handle(Object(Laravel\\Horizon\\Events\\JobFailed))

#11 /var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(366): call_user_func_array(Array, Array)

#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(196): Illuminate\\Events\\Dispatcher->Illuminate\\Events\\{closure}('Laravel\\\\Horizon...', Array)

#13 /var/www/html/vendor/laravel/horizon/src/Listeners/MarshalFailedEvent.php(44): Illuminate\\Events\\Dispatcher->dispatch('Laravel\\\\Horizon...')

#14 [internal function]: Laravel\\Horizon\\Listeners\\MarshalFailedEvent->handle(Object(Illuminate\\Queue\\Events\\JobFailed))

#15 /var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(366): call_user_func_array(Array, Array)

#16 /var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(196): Illuminate\\Events\\Dispatcher->Illuminate\\Events\\{closure}('Illuminate\\\\Queu...', Array)

#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(185): Illuminate\\Events\\Dispatcher->dispatch('Illuminate\\\\Queu...')

#18 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(443): Illuminate\\Queue\\Jobs\\Job->fail(Object(Exception))

#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(430): Illuminate\\Queue\\Worker->failJob(Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Exception))

#20 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(358): Illuminate\\Queue\\Worker->markJobAsFailedIfWillExceedMaxAttempts('redis-sentinel', Object(Illuminate\\Queue\\Jobs\\RedisJob), 1, Object(Exception))

#21 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(331): Illuminate\\Queue\\Worker->handleJobException('redis-sentinel', Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Illuminate\\Queue\\WorkerOptions), Object(Exception))

#22 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(277): Illuminate\\Queue\\Worker->process('redis-sentinel', Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Illuminate\\Queue\\WorkerOptions))

#23 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(118): Illuminate\\Queue\\Worker->runJob(Object(Illuminate\\Queue\\Jobs\\RedisJob), 'redis-sentinel', Object(Illuminate\\Queue\\WorkerOptions))

#24 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(102): Illuminate\\Queue\\Worker->daemon('redis-sentinel', 'low', Object(Illuminate\\Queue\\WorkerOptions))

#25 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(86): Illuminate\\Queue\\Console\\WorkCommand->runWorker('redis-sentinel', 'low')

#26 /var/www/html/vendor/laravel/horizon/src/Console/WorkCommand.php(46): Illuminate\\Queue\\Console\\WorkCommand->handle()

#27 [internal function]: Laravel\\Horizon\\Console\\WorkCommand->handle()

#28 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)

#29 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()

#30 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))

#31 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(576): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)

#32 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\\Container\\Container->call(Array)

#33 /var/www/html/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))

#34 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))

#35 /var/www/html/vendor/symfony/console/Application.php(921): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))

#36 /var/www/html/vendor/symfony/console/Application.php(273): Symfony\\Component\\Console\\Application->doRunCommand(Object(Laravel\\Horizon\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))

#37 /var/www/html/vendor/symfony/console/Application.php(149): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))

#38 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Application.php(90): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))

#39 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(133): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))

#40 /var/www/html/artisan(49): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))

#41 {main}
"} 

When we log the command sent to redis before the exception:

Predis\Command\ZSetAdd::__set_state(array(
   'arguments' => 
(
    0 => 'horizon:0.0.0:failed:api:4054',
    1 => 'HyLbvZ2kbjFlznzM0nHMrOuOF1Ap47zV',
    2 => 'HyLbvZ2kbjFlznzM0nHMrOuOF1Ap47zV',
  ),
))

Is this related to redis sentinel?

[cyrossignol]

Hello @L3o-pold!

This error is not related to Sentinel. In fact, if we review the stack trace, it doesn't look like the application is configured to use Sentinel connections for Horizon because the ZADD command executed through Laravel's standard Redis connection wrapper. If you'd like to use this package for Horizon, please read this section for configuration instructions.

Because the application isn't resolving Horizon's RedisQueue implementation, it generates the string job ID shown in the debug message (HyLbvZ2kbjFlznzM0nHMrOuOF1Ap47zV) instead of an integer. The error occurs because Horizon tries to execute the ZADD command using the string ID for the "score" argument that Redis expects a number for.

When the application is configured to use this package's connections for Horizon (via the redis-sentinel queue connection), it binds the appropriate RedisQueue implementation needed to generate numeric job IDs for Horizon.

[L3o-pold]

Hi @cyrossignol, thanks for your quick reply. That's what I was thinking but it should use redis-sentinel driver.

My .env file contains:

HORIZON_DRIVER=redis-sentinel
CACHE_DRIVER=redis-sentinel
QUEUE_DRIVER=redis-sentinel
QUEUE_CONNECTION=redis-sentinel
REDIS_DRIVER=redis-sentinel

My config/horizon.php contains

'driver' => env('HORIZON_DRIVER', 'redis'),
'use' => 'queue',
'waits' => [
        'redis-sentinel:queue' => 60,
        'redis-sentinel:redis-sentinel' => 60,
        'redis-sentinel:default' => 60,
    ],
'environments' => [
  'development' => [
            'supervisor-1' => [
                'connection'            => 'redis-sentinel',
                'queue'                 => ['high', 'medium', 'default', 'low', 'heavy'],
                'balance'               => 'auto',
                'processes'             => 96,
                'min-processes'         => 0,
                'tries'                 => 1,
                'timeout'               => 3600,
                'memory'                => 32,
                'max-process-for-queue' => [
                    'heavy' => 5,
                ]
            ],
        ],
],

My config/queue.php contains:

'default' => env('QUEUE_CONNECTION', 'sync'),
'connections' => [
  'redis-sentinel' => [
            'driver' => 'redis-sentinel',
            'connection' => 'queue',
            'queue' => 'default',
            'retry_after' => 3666,
            'expire' => 3666,
        ],
],

My config/database.php contains

'redis' => [
        'client'   => env('REDIS_CLIENT', 'predis'),
        'queue'    => [
            'host'     => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD', null),
            'port'     => env('REDIS_PORT', 6379),
            'database' => env('REDIS_DB', 0),
            'cluster'  => 'redis',
        ],

Maybe I'm missing something in the configuration...

[L3o-pold]

An important point is that the job is pushed from a other project (not with a Laravel Framework) but using some Illuminate package for the queue and the laravel-redis-sentinel package.

When pushing a job from the Laravel project (the one who's executing the queue), if the job failed, there is no Predis error.

But when the job is pushed from the other project, if the job failed, there is the Predis error.

Because it's not a Laravel project (so there is no horizon in it) it doesn't push information in the redis with the key horizon:job_id or horizon:recent_jobs. Maybe it came from here.

[cyrossignol]

An important point is that the job is pushed from a other project (not with a Laravel Framework) but using some Illuminate package for the queue and the laravel-redis-sentinel package.

@L3o-pold, you're right--that is an important hint. When the job is pushed from the non-Laravel project (without Horizon), it sets the job ID to the string value shown above instead of to a number as Horizon expects it to be. The base RedisQueue implementation in illuminate/queue is not Horizon-aware, and this package cannot re-bind Horizon's implementation unless the package is available.

I think it's possible to install Horizon without Laravel, but I never tried this myself. You can also extend Laravel's RedisQueue to override the createPayloadArray() method to replace the the job ID generation logic with an approach that's compatible with Horizon. Then, extend Laravel's base queue RedisConnector class to instantiate the custom RedisQueue and reconfigure the QueueManager of the non-Laravel application to use this class instead for Sentinel connections.

An easier approach could be to just register a callback for job serialization that replaces the job ID. You can try adding something like this to the bootstrap process:

use Illuminate\Queue\Queue;
...
Queue::createPayloadUsing(function ($connection, $queue, $payload) {
    if ($connection === 'redis-sentinel') {
        $payload['id'] = // generate horizon job ID...
    }

    return $payload;
});

...with the caveat that this hook system seems to be used for Laravel's test suite, so I'm not sure how well-supported the approach will be in the future.

[L3o-pold]

I think we cannot use the createPayloadUsing method for editing the id. It's generated later...

[L3o-pold]

We extended the RedisQueue and it works perfectly. Thanks for all those advice ❤️

[cyrossignol]

@L3o-pold You're welcome! I'm glad it works.