Redis queue : "Error while reading line from the server"

[trompx]

Hello,

First I am not sure this is a lumen specific error or a Laravel error too. I have an app running on laravel (5.1) which dispatch jobs, handled by a lumen (5.1) worker.

I have multiple jobs running totally fine, however, if a job runs more than 60 seconds (I made some tests 59.75727891922s -> no error, 60.091041898727s -> error), I get the following :

`[2016-02-05 02:27:07] lumen.ERROR: exception 'Predis\Connection\ConnectionException' with message 'Error while reading line from the server. [tcp://127.0.0.1:6379]' in /var/www/app/vendor/predis/predis/src/Connection/AbstractConnection.php:168 Stack trace:

0 /var/www/app/vendor/predis/predis/src/Connection/StreamConnection.php(210): Predis\Connection\AbstractConnection->onConnectionError('Error while rea...')

1 /var/www/app/vendor/predis/predis/src/Connection/AbstractConnection.php(133): Predis\Connection\StreamConnection->read()

2 /var/www/app/vendor/predis/predis/src/Connection/AbstractConnection.php(125): Predis\Connection\AbstractConnection->readResponse(Object(Predis\Command\ZSetRemove))

3 /var/www/app/vendor/predis/predis/src/Client.php(326): Predis\Connection\AbstractConnection->executeCommand(Object(Predis\Command\ZSetRemove))

4 /var/www/app/vendor/predis/predis/src/Client.php(310): Predis\Client->executeCommand(Object(Predis\Command\ZSetRemove))

5 /var/www/app/vendor/illuminate/queue/RedisQueue.php(154): Predis\Client->__call('zrem', Array)

6 /var/www/app/vendor/illuminate/queue/RedisQueue.php(154): Predis\Client->zrem('queues:processi...', '{"job":"Illumin...')

7 /var/www/app/vendor/illuminate/queue/Jobs/RedisJob.php(73): Illuminate\Queue\RedisQueue->deleteReserved('processing', '{"job":"Illumin...')

8 /var/www/app/vendor/illuminate/queue/CallQueuedHandler.php(46): Illuminate\Queue\Jobs\RedisJob->delete()

9 /var/www/app/vendor/illuminate/queue/Jobs/Job.php(129): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\RedisJob), Array)

10 /var/www/app/vendor/illuminate/queue/Jobs/RedisJob.php(51): Illuminate\Queue\Jobs\Job->resolveAndFire(Array)

11 /var/www/app/vendor/illuminate/queue/Worker.php(208): Illuminate\Queue\Jobs\RedisJob->fire()

12 /var/www/app/vendor/illuminate/queue/Worker.php(159): Illuminate\Queue\Worker->process('redis', Object(Illuminate\Queue\Jobs\RedisJob), '3', 0)

13 /var/www/app/vendor/illuminate/queue/Worker.php(111): Illuminate\Queue\Worker->pop(NULL, 'processing', 0, '3', '3')

14 /var/www/app/vendor/illuminate/queue/Worker.php(87): Illuminate\Queue\Worker->runNextJobForDaemon(NULL, 'processing', 0, '3', '3')

15 /var/www/app/vendor/illuminate/queue/Console/WorkCommand.php(103): Illuminate\Queue\Worker->daemon(NULL, 'processing', 0, 128, '3', '3')

16 /var/www/app/vendor/illuminate/queue/Console/WorkCommand.php(71): Illuminate\Queue\Console\WorkCommand->runWorker(NULL, 'processing', 0, 128, true)

17 [internal function]: Illuminate\Queue\Console\WorkCommand->fire()

18 /var/www/app/vendor/illuminate/container/Container.php(502): call_user_func_array(Array, Array)

19 /var/www/app/vendor/illuminate/console/Command.php(150): Illuminate\Container\Container->call(Array)

20 /var/www/app/vendor/symfony/console/Command/Command.php(259): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

21 /var/www/app/vendor/illuminate/console/Command.php(136): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

22 /var/www/app/vendor/symfony/console/Application.php(878): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

23 /var/www/app/vendor/symfony/console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

24 /var/www/app/vendor/symfony/console/Application.php(126): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

25 /var/www/app/vendor/laravel/lumen-framework/src/Console/Kernel.php(78): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

26 /var/www/app/artisan(35): Laravel\Lumen\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

27 {main}

`

I stumble upon this issue https://github.com/nrk/predis/issues/121 but that wasn't that.

I have set an expire time of 120 seconds (https://github.com/laravel/framework/issues/8577), tried to add the 'read_write_timeout' => -1 parameter (http://stackoverflow.com/questions/11776029/predis-is-giving-error-while-reading-line-from-server) in database.php but nothing works.

For the record, some of the jobs use PhpRedis (no persistent connection) so I don't know if that may conflict with Predis (which seems not as all jobs running under 60s don't trigger the error).

My worker config in supervisor is the following : [program:queue_processing] command=php artisan queue:work --daemon --sleep=3 --queue=processing --tries=3 process_name=%(program_name)s_%(process_num)02d numprocs=4 autostart=true autorestart=true

Let me know if I can provide more infos.

[GrahamCampbell]

It doesn't look like an issue with laravel, rather either with your server, or the packages we're using to communicate with redis.

[HashmatWaziri]

try this $redis = $app->make('redis'); // I am using this in route $redis->set('stats', 'd');

[rbdoer]

same problem. any updates?

[shmuelgutman]

I encounter this problem also, for me it's happening while I'm trying the example from https://laravel.com/docs/5.5/redis#pubsub after a few minutes of listening I get the error Error while reading line from the server. [tcp://127.0.0.1:6379] and the process terminated.

[ivanpeter]

I had the same problem. Solved it by installing redis-server on ubuntu, and then starting it using redis-cli ping

1. sudo apt-get install redis-tools
2. sudo apt-get install redis-server
3. redis-cli ping Hope this will be helpful

[freeman3s]

@shmuelgutman I had the same problem with Laravel 5.4. Have you solve it?

[freeman3s]

@shmuelgutman I have fixed it! I've added 'read_write_timeout' => 0 in config/database.php

[maokeyang]

@freeman3s i have added,but nothing works. something not to do?

[freeman3s]

@maokeyang strange, it helped me 'read_write_timeout' => 0

[robinvdvleuten]

You could also try to set the value to -1 to completely disable the timeout as mentioned by the author of Predis https://github.com/nrk/predis/issues/33#issuecomment-1395652. This was the case for me as I was running Laravel Horizon as a daemon script on the server.

[jezmck]

You may also need to ensure you have the scheme set to tls if appropriate.

[craigy-waigy]

@shmuelgutman I have fixed it! I've added 'read_write_timeout' => 0 in config/database.php

'redis' => [

    'client' => env('REDIS_CLIENT', 'predis'),

    'options' => [
        'cluster' => env('REDIS_CLUSTER', 'redis'),
        'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
    ],

    'default' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_DB', '0'),
        'read_write_timeout' => 0,
    ],

[nzkozar]

An important thing to note about setting the read_write_timeout is that it won't have any effect when your required timeout is longer than 300s. This is usually the default Redis server connection timeout configuration.

As mentioned in this issue response: https://github.com/predis/predis/issues/33#issuecomment-1395652

If you are using a managed Redis hosting, check with your provider on how to change this setting. For example Digitalocean allows you to configure this setting for your Redis instances via their API or via support ticket.

In my use cases, Laravel jobs were running well beyond the 300s default timeout value of my Digitalocean Redis instance. Such jobs were failing with the same exception message as the OP posted.

By setting a higher value for the Redis instance timeout config, there was no need to change the Laravel redis connection parameters.