Using Redis Clusters

[patrickguevara]

This might be a misunderstanding from the docs, but I need some assistance when setting up the config for Horizon/queues.

config/database.php

    'redis' => [
        'client' => 'predis',
        'options' => [
            'cluster' => 'redis',
            'parameters' => [
                'password' => env('REDIS_PASSWORD', null),
                'scheme' => env('REDIS_SCHEME', 'tcp'),
            ],
        ],
        'clusters' => [
            'default' => [
                [
                    'host' => env('REDIS_HOST', '127.0.0.1'),
                    'password' => env('REDIS_PASSWORD', null),
                    'port' => env('REDIS_PORT', 6379),
                    'database' => 0,
                    'read_write_timeout' => 60,
                ],
            ],
        ],
    ],

config/queue.php

    'connections' => [
        'sync' => [
            'driver' => 'sync',
        ],
        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => '{default}',
            'retry_after' => 90,
        ],
    ],

When I ran composer require laravel/horizon I got the following during the artisan package:discover:

In Horizon.php line 91:
  Redis connection [default] has not been configured.

1. Does Horizon work with clustered Redis?
2. Do I need to define default alongside clusters in database.php?

[patrickguevara]

This can be fixed by setting config.horizon.use to clusters.default in config/horizon.php,

return [
    'use' => 'clusters.default',
];

But then I get Cannot use 'EVAL' with redis-cluster. after the queued Job runs successfully.

In laravel.log:

[2018-01-11 12:40:08] production-alpha.ERROR: Cannot use 'EVAL' with redis-cluster. {"exception":"[object] (Predis\\NotSupportedException(code: 0): Cannot use 'EVAL' with redis-cluster. at /www/mediapp/vendor/predis/predis/src/Connection/Aggregate/RedisCluster.php:380)
[stacktrace]
#0 /www/mediapp/vendor/predis/predis/src/Connection/Aggregate/RedisCluster.php(550): Predis\\Connection\\Aggregate\\RedisCluster->getConnection(Object(Predis\\Command\\ServerEval))
#1 /www/mediapp/vendor/predis/predis/src/Connection/Aggregate/RedisCluster.php(593): Predis\\Connection\\Aggregate\\RedisCluster->retryCommandOnFailure(Object(Predis\\Command\\ServerEval), 'executeCommand')
#2 /www/mediapp/vendor/predis/predis/src/Client.php(331): Predis\\Connection\\Aggregate\\RedisCluster->executeCommand(Object(Predis\\Command\\ServerEval))
#3 /www/mediapp/vendor/predis/predis/src/Client.php(314): Predis\\Client->executeCommand(Object(Predis\\Command\\ServerEval))
#4 /www/mediapp/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(96): Predis\\Client->__call('eval', Array)
#5 /www/mediapp/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(108): Illuminate\\Redis\\Connections\\Connection->command('eval', Array)
#6 /www/mediapp/vendor/laravel/horizon/src/Repositories/RedisMetricsRepository.php(200): Illuminate\\Redis\\Connections\\Connection->__call('eval', Array)
#7 /www/mediapp/vendor/laravel/horizon/src/Listeners/UpdateJobMetrics.php(53): Laravel\\Horizon\\Repositories\\RedisMetricsRepository->incrementQueue('{default}', 2199.59)
#8 [internal function]: Laravel\\Horizon\\Listeners\\UpdateJobMetrics->handle(Object(Laravel\\Horizon\\Events\\JobDeleted))
#9 /www/mediapp/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(369): call_user_func_array(Array, Array)
#10 /www/mediapp/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(200): Illuminate\\Events\\Dispatcher->Illuminate\\Events\\{closure}('Laravel\\\\Horizon...', Array)
#11 /www/mediapp/vendor/laravel/horizon/src/RedisQueue.php(155): Illuminate\\Events\\Dispatcher->dispatch('Laravel\\\\Horizon...')
#12 /www/mediapp/vendor/laravel/horizon/src/RedisQueue.php(124): Laravel\\Horizon\\RedisQueue->event('{default}', Object(Laravel\\Horizon\\Events\\JobDeleted))
#13 /www/mediapp/vendor/laravel/framework/src/Illuminate/Queue/Jobs/RedisJob.php(84): Laravel\\Horizon\\RedisQueue->deleteReserved('{default}', Object(Illuminate\\Queue\\Jobs\\RedisJob))
#14 /www/mediapp/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(57): Illuminate\\Queue\\Jobs\\RedisJob->delete()
#15 /www/mediapp/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(76): Illuminate\\Queue\\CallQueuedHandler->call(Object(Illuminate\\Queue\\Jobs\\RedisJob), Array)
#16 /www/mediapp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(320): Illuminate\\Queue\\Jobs\\Job->fire()
#17 /www/mediapp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(270): Illuminate\\Queue\\Worker->process('redis', Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Illuminate\\Queue\\WorkerOptions))
#18 /www/mediapp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(114): Illuminate\\Queue\\Worker->runJob(Object(Illuminate\\Queue\\Jobs\\RedisJob), 'redis', Object(Illuminate\\Queue\\WorkerOptions))
#19 /www/mediapp/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\\Queue\\Worker->daemon('redis', '{default}', Object(Illuminate\\Queue\\WorkerOptions))
#20 /www/mediapp/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(85): Illuminate\\Queue\\Console\\WorkCommand->runWorker('redis', '{default}')
#21 [internal function]: Illuminate\\Queue\\Console\\WorkCommand->handle()
#22 /www/mediapp/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#23 /www/mediapp/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#24 /www/mediapp/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#25 /www/mediapp/vendor/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#26 /www/mediapp/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\\Container\\Container->call(Array)
#27 /www/mediapp/vendor/symfony/console/Command/Command.php(252): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#28 /www/mediapp/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#29 /www/mediapp/vendor/symfony/console/Application.php(938): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#30 /www/mediapp/vendor/symfony/console/Application.php(240): Symfony\\Component\\Console\\Application->doRunCommand(Object(Laravel\\Horizon\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#31 /www/mediapp/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#32 /www/mediapp/vendor/laravel/framework/src/Illuminate/Console/Application.php(88): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#33 /www/mediapp/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#34 /www/mediapp/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#35 {main}
"}

Any ideas?

[maxwellimpact]

I'm seeing this as well. Going to dig in a bit more to see why. I'll post back if I find a fix.

[maxwellimpact]

I ended up reverting to a non-cluster based approach for now. It appears clusters work fine for cache, but there are still some gotchas with using it for queues.

Based on the comments from these issues there are ways around it, but it doesn't appear to be very straight forward and likely means changes need to be made to the core via a PR. https://github.com/nrk/predis/issues/405 https://github.com/laravel/framework/issues/18527

[justinwilson-apollidon]

Any updates on this? I have experienced this as well and was unable to get Horizon working with an AWS Redis Cluster (ElastiCache).

[patrickguevara]

I ended up using redis with cluster mode disabled. Our use case was such that the clustered redis did not offer any material advantages over the non-clustered instance. You still get high availability with automatic AZ failover without clustering. We were just paying 3x the price for no added benefit.

Another note, Horizon (or maybe Predis) was not utilizing a majority of the nodes/shards. We had a cluster set up to only handle the Horizon queue and out of the 9 nodes (3 shards, 2 replicas each) we were only getting CacheHits on 1 node. Every other node (8 nodes!) was at 0 CacheHits/CacheMisses.

I would seriously consider whether you need clustered redis before committing to it. In our case it was just an expensive headache. Since we moved to non-clustered instances we're saving a lot of money with a much simpler configuration.

[justinwilson-apollidon]

@patrickguevara Fair enough, we actually did the same as well. Horizon seems to work fine without redis in cluster mode, though it would be nice that Laravel Horizon not claim to be able to do so, if it indeed cannot. Just my two cents, though.

[toopay]

It seems to me that these lines at Horizon.php will always prevent RedisManager.php to resolve cluster connection. From RedisManager

if (isset($this->config[$name])) {
     return $this->connector()->connect($this->config[$name], $options);
}

Since the config merge happening in the Horizon.php, it won't ever reach the next condition, which will enable cluster :

if (isset($this->config['clusters'][$name])) {
       return $this->resolveCluster($name);
}

I am not digging furthermore, but from this logic chain alone, Horizon will always use non-cluster connection. cc @themsaid

[toopay]

With small changes on Horizon.php, and some adjustment on the usage of {} (hashtag, to ensure multiple keys are stored in a same hash slot) in RedisMetricRepository.php I get horizon working in redis cluster mode. @themsaid is this something you're interesting to support? If so, I can prepare a PR.

[support-sysagents]

@toopay would be great if you could do that and send a PR .

We are facing the same issue here

[adzon]

@toopay would be very great if you could do that and send a PR .

We are facing the same issue here

[klipitkas]

I am not sure why but it appears that Horizon searches for a cluster named 'horizon' by default so if you have the following configuration inside the database.php it appears to work:

    /*
    |--------------------------------------------------------------------------
    | Redis Databases
    |--------------------------------------------------------------------------
    |
    | Redis is an open source, fast, and advanced key-value store that also
    | provides a richer set of commands than a typical key-value systems
    | such as APC or Memcached. Laravel makes it easy to dig right in.
    |
    */

    'redis' => [

        'client' => 'predis',

        'options' => [
            'cluster' => 'redis'
        ],

        'clusters' => [
            'default' => [
                [
                    'host' => env('REDIS_HOST', 'localhost'),
                    'password' => env('REDIS_PASSWORD', null),
                    'port' => env('REDIS_PORT', 6379),
                    'database' => 0,
                ],
            ],
            'horizon' => [
                [
                    'host' => env('REDIS_HOST', 'localhost'),
                    'password' => env('REDIS_PASSWORD', null),
                    'port' => env('REDIS_PORT', 6379),
                    'database' => 0,
                ],
            ],
        ],
    ],

Inside the clusters I have added the horizon cluster configuration which makes it work. Also inside the horizon.php file I have the following configuration:

    /*
    |--------------------------------------------------------------------------
    | Horizon Redis Connection
    |--------------------------------------------------------------------------
    |
    | This is the name of the Redis connection where Horizon will store the
    | meta information required for it to function. It includes the list
    | of supervisors, failed jobs, job metrics, and other information.
    |
    */

    'use' => 'clusters.default',

[flexgrip]

Tried this recent suggested config from @klipitkas. Does not work. Still complains about Cannot use 'EVAL' with redis-cluster.

Is there anything else I would have to add to get this to work? I've changed the settings in both files.

[taylorotwell]

Horizon does not work with Redis Cluster and likely never will. Even if it did work with Clusters, you wouldn't be taking advantage of the clustering because Lua operations can't operate on multiple shards in one operation. And, for Horizon to work properly, our Lua scripts have to perform operations on multiple keys in one atomic transaction.

[abbood]

@taylorotwell does laravel pub/sub work with redis clusters? see my question here: https://stackoverflow.com/questions/56531037/cannot-use-pub-sub-with-redis-cluster

[aromka]

@taylorotwell - It's been almost 5 years since this comment. Has anything changed since then? Since more companies offer Redis Serverless more and more, which requires clustering, it's a must have at this point to support clusters. Is it something on the roadmap / in works?

[daison12006013]

if anyone stumble upon this issue, you might wanna try out my modified version that supports redis-cluster

daison12006013/laravel-horizon-cluster

[ajnozari]

The fact that this is still an issue 6 years later from the original 2018 issue is ridiculous. Redis Cluster isn’t going away and horizon needs to be updated to support this.

[GTCrais]

Subscribing to this thread, in case this is ever supported.