Struggling to get Queues working with redis sentinel

[msencenb]

Hey!

First - thanks for writing a redis sentinel driver, much appreciated. I'm deploying this driver alongside an upgrade to laravel 5.5, and my infrastructure uses a group of 3 sentinels.

I seem to be connected to laravel with this driver ok, as running (string)app('redis-sentinel')->connection()->ping() equals 'PONG'

But my queues, which are run via supervisor, are no longer processing. Relevant code pasted below:

Supervisor conf template:

[program:text-message-queue]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work redis-sentinel --queue=text_message --sleep=5 --tries=7 --env=production
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/logs/worker.log

[program:order-callback-queue]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work redis-sentinel --queue=order_callback --sleep=5 --tries=7 --env=production
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/adopterluv/storage/logs/worker.log

[program:email-queue]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work redis-sentinel --queue=email --sleep=3 --tries=7 --env=production
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/logs/worker.log

[group:laravel-workers]
programs=text-message-queue,order-callback-queue,email-queue

And my database config looks like this:

'redis-sentinel' => [
    'default' => [
        [
            'host' => env('REDIS_HOST', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        [
            'host' => env('REDIS_HOST_2', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        [
            'host' => env('REDIS_HOST_3', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        'options' => [
            'service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'),
            'parameters' => [
                'password' => env('REDIS_PASSWORD', null),
                'database' => 0,
            ],
        ],
    ],

    'cache' => [
        [
            'host' => env('REDIS_HOST', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        [
            'host' => env('REDIS_HOST_2', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        [
            'host' => env('REDIS_HOST_3', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        'options' => [
            'service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'),
            'parameters' => [
                'password' => env('REDIS_PASSWORD', null),
                'database' => 1,
            ],
        ],
    ],

    'session' => [
        [
            'host' => env('REDIS_HOST', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        [
            'host' => env('REDIS_HOST_2', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        [
            'host' => env('REDIS_HOST_3', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        'options' => [
            'service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'),
            'parameters' => [
                'password' => env('REDIS_PASSWORD', null),
                'database' => 2,
            ],
        ],
    ],

    'queue' => [
        [
            'host' => env('REDIS_HOST', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        [
            'host' => env('REDIS_HOST_2', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        [
            'host' => env('REDIS_HOST_3', 'localhost'),
            'port' => env('REDIS_PORT', 26379)
        ],
        'options' => [
            'service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'),
            'parameters' => [
                'password' => env('REDIS_PASSWORD', null),
                'database' => 3,
            ],
        ],
    ],
],

My connections array in queue.php has this entry:

'redis-sentinel' => [
    'driver' => 'redis-sentinel',
    'connection' => 'queue',
    'queue' => 'default',
    'retry_after' => 90, // Laravel >= 5.4.30
],

The env variable QUEUE_DRIVER is set to redis-sentinel. I enqueue various jobs like this to the different queues:

$job = (new SendCheckoutTextMessage($this, $adapter))->onQueue('text_message');
$this->dispatch($job);

Have you had an issues getting queues working?

[msencenb]

Hmm, actually this appears to be working just fine now.

My supervisor process had been killed on one of the web servers that runs the supervisor group. Restarting my whole fleet healed everything up and restarted those processes again. Sorry for the issue pollution!

[cyrossignol]

Nice when things fix themselves! Glad you got this working... now I just need to tag the dev branch so you 5.5ers can use the fancy autoloading 🙂

To save the environment, do you mind if I recycle this ticket to ask for your opinion about this package's environment-based config? I wanted to simplify the giant config entries (you probably noticed them!) and modifications to multiple config files that we needed to write every time in new projects. We can set up your example without modifying any default config files by assigning the env vars shown in the Quickstart (almost verbatim).

It's a new feature, so I'm curious to know how projects use it and what limitations might prevent its adoption. Maybe I just need to clear up the docs. Is there a reason you chose to use standard config files for your project?

Thanks!

[msencenb]

Hmmm I hadn't even noticed that autoloading wasn't tagged yet. Does that only affect me if I'm calling redis-sentinel directly via the facade?

To be honest I had originally planned on using the environment based config but I kept running in to snags in my infrastructure, particularly when switching between local and my pre production environments. Between my local homestead (single redis), my pre-prod environment that runs my develop branch (which uses a single redis), and my second pre-prod environment that runs my release branches (which has sentinels) I had a lot to keep straight.

My answer to all that confusion was to drop down to the very precise configuration to get rid of the 'magic' of the environment based setup. The environment based configuration was hard to debug as I wasn't sure if I had screwed up the config or it was something else in the stack.

At the end of the day the snags had nothing to do with this excellent driver, but now that I have painstakingly created my big configs I'm probably going to stick with them to make it very clear to future devs on my team what's going on.

Btw - the documentation was great, kudos for being vigilant with that!

[cyrossignol]

Hmmm I hadn't even noticed that autoloading wasn't tagged yet. Does that only affect me if I'm calling redis-sentinel directly via the facade?

Currently, the package won't bind to the framework if your project doesn't register the service provider in config/app.php. I'll get autoloading tagged and bagged this weekend.

I stalled to do some thinking about it....and decided not to autoload the package's facade because a PR for the PhpRedis extension (also this one) may introduce a RedisSentinel class in the global namespace. This package will eventually support that extension as a Redis client option, so I want to avoid unexpected naming ambiguity. If PhpRedis rejects that PR or changes the name of that class, we can always include facade auto-aliasing later without introducing a breaking change and confusion.

...I kept running in to snags in my infrastructure, particularly when switching between local and my pre production environments.

Ironically, I hoped that environment-based config would help mitigate some of these cross-environment uncertainties! I have some ideas (including an artisan command) that improve configuration transparency and help troubleshoot complex Sentinel setups. Your experience here motivates me to start building this functionality.

...but now that I have painstakingly created my big configs I'm probably going to stick with them to make it very clear to future devs on my team what's going on.

Very reasonable--I understand completely. Thanks for the kind feedback!

[msencenb]

Ahh now I understand why I was confused by the auto loading, I had explicitly added it to my config. That mystery is solved!

[cyrossignol]

@msencenb Autoloading released in version 2.3.0!