Closed azhard4int closed 6 years ago
Hi @azhard4int, Sentinel servers typically listen on port 26379 (I'll update tests/stubs/config.php
to reflect this, thanks!). Check the config file used to start your Sentinel processes to confirm the port number.
Try running redis-cli -h your-sentinel-host -p 26379 ping
to verify that Sentinel is configured properly and accepting connections on that port. If you don't see PONG
, there's a problem with the Sentinel server's configuration or connectivity.
The error message in the title seems to indicate that the redis-sentinel
driver in config/database.php
doesn't contain a Sentinel connection with at least one host. I can try to offer more useful help if you provide that part of the config file.
@cyrossignol For the ping response:
> redis-cli -h 45.76.XX.XX -p 26379 ping
> PONG
Here is the config file
<?php
return [
'fetch' => PDO::FETCH_OBJ,
'default' => env('DB_CONNECTION', 'mongodb'),
'connections' => [
'sqlite' => [
'driver' => 'sqlite',
'database' => env('DB_DATABASE', database_path('database.sqlite')),
'prefix' => '',
],
'mysql' => [
'driver' => 'mysql',
'host' => env('DB_HOST', 'localhost'),
'port' => env('DB_PORT', '3306'),
'database' => env('DB_DATABASE', 'forge'),
'username' => env('DB_USERNAME', 'forge'),
'password' => env('DB_PASSWORD', ''),
'charset' => 'utf8',
'collation' => 'utf8_unicode_ci',
'prefix' => '',
'strict' => true,
'engine' => null,
],
'pgsql' => [
'driver' => 'pgsql',
'host' => env('DB_HOST', 'localhost'),
'port' => env('DB_PORT', '5432'),
'database' => env('DB_DATABASE', 'forge'),
'username' => env('DB_USERNAME', 'forge'),
'password' => env('DB_PASSWORD', ''),
'charset' => 'utf8',
'prefix' => '',
'schema' => 'public',
'sslmode' => 'prefer',
],
'default' => [
'host' => env('REDIS_HOST', 'tcp://127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
],
'migrations' => 'migrations',
'redis' => [
'cluster' => false,
'default' => [
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
'options' => [
'parameters' => ['password' => env('REDIS_PASSWORD', null)],
],
],
'redis-sentinel' => [
'default' => [
[
'host' => '45.76.XX.XX',
'port' => 26379
],
[
'host' => '45.32.XX.XX',
'port' => 26379
],
[
'host' => '45.32.XX.XX',
'port' => 26379
]
],
'options' => [
'service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'),
'parameters' => [
'password' => env('REDIS_PASSWORD', null),
'database' => 0,
],
],
],
];
Thanks, @azhard4int. Your configuration looks fine. How are you executing the Redis GET
and SET
commands? I can't tell for sure, but it seems like the app is using the redis
connection instead of the redis-sentinel
connection. The error in the issue title indicates that the Predis client cannot find any Sentinel hosts to query, because they don't exist in the config, or because PHP can't connect. Please try this to check the config that the package loaded, and post back if you don't see a problem:
var_dump(app('redis-sentinel'));
If you're using the Redis
facade, you'll need to set 'redis' => [ ... 'driver' => 'sentinel', ... ]
in config/database.php
so the facade uses the Sentinel connection (see overriding the standard Redis API).
You can also use the RedisSentinel
facade if you don't want to override Laravel's standard Redis API.
If you're on a slow network, you might try increasing the amount of time that the client waits for a response from a Sentinel server:
'redis-sentinel' => [
...
'options' => [
...
'sentinel_timeout' => 0.100, // default (seconds)
]
]
You can remove the out-of-place 'default'
element in the 'connections'
array in that config as well as the 'options'
element in the 'redis'
array. These probably are not causing your issue, just a clean-up.
@cyrossignol thank you for the detailed response.
It was indeed the problem with the network connection, I have increased the sentinel timeout and it works fine now.
One more thing I have noticed is, I have redis-sentinel in the connections array for queue.php, but it picks the default non-sentinel redis drivers.
Below is the connections array:
'connections' => [
'sync' => [
'driver' => 'sync',
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
],
'beanstalkd' => [
'driver' => 'beanstalkd',
'host' => 'localhost',
'queue' => 'default',
'retry_after' => 90,
],
'sqs' => [
'driver' => 'sqs',
'key' => 'your-public-key',
'secret' => 'your-secret-key',
'prefix' => 'https://sqs.us-east-1.amazonaws.com/your-account-id',
'queue' => 'your-queue-name',
'region' => 'us-east-1',
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'retry_after' => 90,
],
'redis-sentinel' => [
'driver' => 'redis-sentinel',
'connection' => 'default',
'queue' => 'default',
'expire' => 60,
],
],
And, If I change the driver for the redis key driver value to redis-sentinel it works with the redis sentinel.
Should not it be picking the redis-sentinel key instead of the redis key?
I'm glad you solved the connection issue. To answer your next question:
The config/queue.php
file usually contains a 'default'
element near the top that tells Laravel which queue connection to use:
'default' => env('QUEUE_DRIVER', 'sync'),
Check the value of QUEUE_DRIVER
in .env
. You may need to change it from redis
to redis-sentinel
.
@cyrossignol I have redis-sentinel in the .env
for QUEUE_DRIVER
.
Also, I have tried changing the 'default'=> env('QUEUE_DRIVER', 'sync'),
value directly to the 'default'=>'redis-sentinel'
but it is still picking default redis connection.
If you're setting the literal value of 'default'
to 'redis-sentinel'
, then something else must be changing it down the line. You might try looking for any Config::set('queue.default', 'redis')
or Queue::setDefaultDriver('redis')
calls. What does Queue::getDefaultDriver()
show right before your queue logic runs?
@cyrossignol it was onConnection('redis') that was causing this issue.
Thanks a lot man, really appreciate it.
I have the same problem. It happens only in queued jobs. What could be the reason ?
Queue::getDefaultDriver() is correctly set to 'redis-sentinel'
Here is my config/database key for redis-sentinel
'redis-sentinel' => [
'default' => [
[
'host' => env('REDIS_HOST', 'localhost'),
'port' => env('REDIS_PORT', 26379),
],
],
'options' => [
'service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'),
'parameters' => [
'password' => env('REDIS_PASSWORD', null),
'database' => 0,
],
'sentinel_timeout' => 0.251,
'update_sentinels' => true,
],
],
Hi @bdteo,
Do you see the error for every queued job? Or only some of the queued jobs? We can debug the state of the Sentinel connection by adding this temporary code to the boot()
method of a service provider like AppServiceProvider
:
\Queue::before(function ($event) {
$lines = [ 'Connection: ' . $event->connectionName ];
$dumper = new \Symfony\Component\VarDumper\Dumper\CliDumper();
$dumper->setOutput(function ($line) use (&$lines) {
$lines[] = $line;
});
$queue = $event->job->getRedisQueue();
$method = (new \ReflectionObject($queue))->getMethod('getConnection');
$method->setAccessible(true);
$dumper->dump((new \Symfony\Component\VarDumper\Cloner\VarCloner())
->cloneVar($method->invoke($queue)->client()->getConnection());
\Log::debug(implode(PHP_EOL, $lines));
});
Edit: fixed for protected RedisQueue::getConnection()
method.
This will write the state of the Sentinel connection to storage/logs/laravel.log before each job runs so we can verify that it contains the Sentinel servers defined in the REDIS_HOST
environment variable. Then, queue up a job for processing and check the output in the log. If you're not sure how to read the output, please post it back here.
Ok, thank you very much for the helpful suggestion.
It seems that for my composer setup the getConnection
method of RedisQueue
is protected so this is how I used your code ....
Queue::before(function ($event) {
$lines = [ 'Connection: ' . $event->connectionName ];
$dumper = new \Symfony\Component\VarDumper\Dumper\CliDumper();
$dumper->setOutput(function ($line) use (&$lines) {
$lines[] = $line;
});
$connectionObjOfInterest = $event->job->getRedisQueue();
$reflector = new ReflectionObject($connectionObjOfInterest);
$method = $reflector->getMethod('getConnection');
$method->setAccessible(true);
$dumper->dump((new \Symfony\Component\VarDumper\Cloner\VarCloner())
->cloneVar($method->invoke($connectionObjOfInterest)->client()->getConnection()));
Log::debug(implode(PHP_EOL, $lines));
});
Here is the result ....
call_stack: NULL
instance: xxxxxxxxxxxx
channel: pre-production
ip_address: 10.156.0.4
level: DEBUG
message: Connection: redis-sentinel
Predis\Connection\Aggregate\SentinelReplication {#1063
#master: Predis\Connection\StreamConnection {#1069
-resource: stream resource {@566
timed_out: false
blocked: true
eof: false
stream_type: "tcp_socket/ssl"
mode: "r+"
unread_bytes: 0
seekable: false
options: []
}
-cachedId: null
#parameters: Predis\Connection\Parameters {#1068
-parameters: array:6 [
"host" => "10.156.0.8"
"port" => "6379"
"alias" => "master"
"password" => "xxxxxxxxxxxx"
"database" => 0
"scheme" => "tcp"
]
}
#initCommands: array:2 [
0 => Predis\Command\RawCommand {#1070
-slot: null
-commandID: "AUTH"
-arguments: array:1 [
0 => "xxxxxxxxxxxx"
]
}
1 => Predis\Command\RawCommand {#1071
-slot: null
-commandID: "SELECT"
-arguments: array:1 [
0 => 0
]
}
]
}
#slaves: []
#current: Predis\Connection\StreamConnection {#1069}
#service: "servicexxxx"
#connectionFactory: Predis\Connection\Factory {#1065
-defaults: array:2 [
"password" => "xxxxxxxxxxxx"
"database" => 0
]
#schemes: array:6 [
"tcp" => "Predis\Connection\StreamConnection"
"unix" => "Predis\Connection\StreamConnection"
"tls" => "Predis\Connection\StreamConnection"
"redis" => "Predis\Connection\StreamConnection"
"rediss" => "Predis\Connection\StreamConnection"
"http" => "Predis\Connection\WebdisConnection"
]
}
#strategy: Predis\Replication\ReplicationStrategy {#1064
#disallowed: array:11 [
...
"INFO" => true
...
"SLOWLOG" => true
]
#readonly: array:62 [
"EXISTS" => true
....
"ECHO" => true
...
"PFCOUNT" => true
"SORT" => array:2 [
0 => Predis\Replication\ReplicationStrategy {#1064}
1 => "isSortReadOnly"
]
"BITFIELD" => array:2 [
0 => Predis\Replication\ReplicationStrategy {#1064}
1 => "isBitfieldReadOnly"
]
"GEOHASH" => true
"GEOPOS" => true
"GEODIST" => true
"GEORADIUS" => array:2 [
0 => Predis\Replication\ReplicationStrategy {#1064}
1 => "isGeoradiusReadOnly"
]
"GEORADIUSBYMEMBER" => array:2 [
0 => Predis\Replication\ReplicationStrategy {#1064}
1 => "isGeoradiusReadOnly"
]
]
#readonlySHA1: []
}
#sentinels: array:3 [
0 => array:6 [
"host" => "10.156.0.8"
"port" => "26379"
"database" => null
"password" => null
"timeout" => 0.5
"scheme" => "tcp"
]
1 => array:2 [
"host" => "10.156.0.9"
"port" => "26379"
]
2 => array:2 [
"host" => "10.156.0.10"
"port" => "26379"
]
]
#sentinelConnection: Predis\Connection\StreamConnection {#1067
-resource: stream resource {@565
timed_out: false
blocked: true
eof: false
stream_type: "tcp_socket/ssl"
mode: "r+"
unread_bytes: 0
seekable: false
options: []
}
-cachedId: null
#parameters: Predis\Connection\Parameters {#1062
-parameters: array:6 [
"host" => "10.156.0.8"
"port" => "26379"
"database" => null
"password" => null
"timeout" => 0.5
"scheme" => "tcp"
]
}
#initCommands: []
}
#sentinelTimeout: 0.5
#retryLimit: 20
#retryWait: 1000
#updateSentinels: true
}
context: []
created_at: 2018-12-20 20:44:45
1 row in set (0.00 sec)
Everything seems to be ok.
I spent almost my whole day reading through the source code of predis
, monospice/laravel-redis-sentinel-drivers
and laravel
.
I came to the suspicion that running the predis
client in a Laravel queue worker context may trigger a dangerous bug. I don't know whether I am right or wrong as of this moment but here is what I mean ....
I found that once I restart my artisan queue:work
supervisord
workers on all servers then everything seems to be ok (until I deploy my production images again).
Basically said my theory is that Laravel preserves single Predis\Connection\Aggregate\SentinelReplication
object for the whole runtime of a queue worker (due to the way Laravel's service container works and the way Laravel workers work).
So ... Single Predis\Connection\Aggregate\SentinelReplication object - single instance of its property
$sentinels
/**
* @var NodeConnectionInterface[]
*/
protected $sentinels = array();
I suppose this is the culprit
/**
* Returns the current sentinel connection.
*
* If there is no active sentinel connection, a new connection is created.
*
* @return NodeConnectionInterface
*/
public function getSentinelConnection()
{
if (!$this->sentinelConnection) {
if (!$this->sentinels) {
throw new \Predis\ClientException('No sentinel server available for autodiscovery.');
}
$sentinel = array_shift($this->sentinels);
$this->sentinelConnection = $this->createSentinelConnection($sentinel);
}
return $this->sentinelConnection;
}
When a connection fails then this happens: $this->sentinelConnection = null;
.
As far as I see this means that if it happens that all of the connections fail at the same time then $sentinels
becomes an empty array for the rest ot the runtime (in a Laravel context).
I am very tired to check this now, so excuse me for my lowered abilities to write reasonable senteces :D I will write tomorrow.
Do you think that this might be the problem ? (I cannot confirm it until I produce the bug forcefully.)
@bdteo Thanks for providing very detailed information. Your configuration appears fine. It looks like you found a bug in Predis, and you correctly identified the culprit:
public function getSentinelConnection()
{
...
$sentinel = array_shift($this->sentinels); // drops the connection forever
...
}
We won't likely see this problem during the short web-request life-cycle because the set of Sentinel connections re-initializes with each request. However, it's just like you said: in a long-running PHP process—like a Laravel queue worker—Predis can eventually drain the set of available Sentinel connections after each connection times-out at some point during the lifetime of the process.
Unfortunately, it doesn't seem like Predis receives regular maintenance right now. I'll start looking into a workaround for this package. In the meantime, try increasing the sentinel_timeout
value for the queue service by adding a dedicated connection in config/database.php:
'redis-sentinel' => [
...
'queue' => [
...
'options' => [
'sentinel_timeout' => 5.0,
],
],
],
...and change the 'connection'
to 'queue'
for 'redis-sentinel'
in the 'connections'
array in config/queue.php.
In many cases, a large timeout value here shouldn't significantly impact the response time of the application because the queue workers run in the background. It looks like you've already set update_sentinels
to true
. This improves the chance that the application will receive a new set of Sentinel connections when Predis drops one. You can also schedule a job to restart the queue workers periodically, or set a custom queue exception handler that restarts workers after catching Predis\ClientException
.
In a well-designed Sentinel deployment, a major outage in which every Sentinel becomes unreachable by a client should rarely happen (if it does, bigger problems exist—this isn't a failure scenario that Sentinel is designed to handle). When Predis loses all of its Sentinel connections, the config likely contains a connection timeout value too low for a given network. Depending on the application's response-time requirement, we can either increase the timeout value or fix the networking issues that cause this latency. That said, this package should handle brief moments of intermittent connectivity or network congestion. I'll start working on a solution as soon as possible.
Thank you for the detailed information that you provided. It will be helpful for sure in future scenarios.
I agree about Sentinels' deployment and the best practices. However this is not always the case. Often there is a single point of failure - a human being who controls the Redis setup.
This is entirely up to company structure, dev ops and so on ...
So a bug is a bug. It is cool that we can improve the predis
project.
I don't have so much knowledge about its general architecture, so can you please submit the bug to their repository :) ?
I got this similar error not able to solve it. I did try to ping all my sentinel server and it is working. below is my configuration:
Laravel version 5.1
"monospice/laravel-redis-sentinel-drivers": "^1.0"
"predis/predis": "1.1"
PHP version 7
I used env file configuration, below is my env file configuration:
CACHE_DRIVER=redis-sentinel SESSION_DRIVER=redis-sentinel QUEUE_DRIVER=redis-sentinel REDIS_DRIVER=redis-sentinel REDIS_HOST=XX.XXX.XXX.XX, XX.XXX.XXX.XX, XX.XXX.XXX.XX REDIS_PORT=26379 REDIS_SENTINEL_DISCOVERY=true REDIS_SENTINEL_SERVICE=mymaster REDIS_SENTINEL_PASSWORD="xxxxxxxxxx" REDIS_SENTINEL_TIMEOUT=0.100
May I know what wrong with my configuration?
Hi @pangminfu,
Try increasing REDIS_SENTINEL_TIMEOUT
. Predis sets the default timeout value to 0.100
seconds, which may be too short in some environments.
As described in this ticket, we tend to see the "No sentinel server available for autodiscovery" error on slow networks. This might happen when Sentinel servers don't run on the same host or physical network as the application (like in some cloud hosting environments).
Otherwise, your configuration looks fine. One minor suggestion: we can change REDIS_SENTINEL_PASSWORD
to REDIS_PASSWORD
because it doesn't look like you're using both Sentinel and Laravel's standard Redis API at the same time. In this case, it works the same either way.
Hi @cyrossignol
Increase the timeout doesn't help. What happen if my remote sentinel server doesn't set any password and i had commented the REDIS_PASSWORD. I checked the sentinel debug log is showing client closed connection. below is my log :
36043:X 07 Jan 12:05:24.398 - Accepted xxx.xxx.xx.xx:58559 36043:X 07 Jan 12:05:24.440 - Client closed connection
36043:X 07 Jan 12:05:24.398 - Accepted xxx.xxx.xx.xx:58559 36043:X 07 Jan 12:05:24.440 - Client closed connection
@pangminfu, these messages are normal to see when we set the Redis server's loglevel
to verbose
or debug
.
Just to confirm--do you see the "No sentinel server available for autodiscovery" exception message exactly? If so, does this exception occur every time the application tries to connect to Redis? Or does it only happen occasionally?
Both REDIS_PASSWORD
and REDIS_SENTINEL_PASSWORD
set the password for the Redis servers monitored by Sentinel if needed (Sentinel doesn't support password authentication itself). Set the second one if the application also needs to use Laravel's regular, non-Sentinel Redis connections.
Hi @cyrossignol ,
I am constantly getting "No sentinel server available for autodiscovery" exception message.
I understand that Sentinel itself doesn't support password authentication but what i mean is my master doesn't set any password.
I also notice that laravel 5.1 is having issue with predis version 1.1 even before add in this package but i manage to solve it according to this solution below : https://github.com/nrk/predis/issues/390#issuecomment-266522624
So I was wonder could this cause "No sentinel server available for autodiscovery"? Also what is the value I should set for REDIS_DATABASE and what does this value mean?
I tried to do debugging seem like the exception thrown is due to error in below code : SentinelReplication.php
protected function querySentinelForSlaves(NodeConnectionInterface $sentinel, $service)
{
$slaves = array();
$payload = $sentinel->executeCommand(
RawCommand::create('SENTINEL', 'slaves', $service)
);
if ($payload instanceof ErrorResponseInterface) {
$this->handleSentinelErrorResponse($sentinel, $payload);
}
foreach ($payload as $slave) {
$flags = explode(',', $slave[9]);
if (array_intersect($flags, array('s_down', 'o_down', 'disconnected'))) {
continue;
}
$slaves[] = array(
'host' => $slave[3],
'port' => $slave[5],
'alias' => "slave-$slave[1]",
);
}
return $slaves;
}
the error happen in below line :
$payload = $sentinel->executeCommand( RawCommand::create('SENTINEL', 'slaves', $service) );
the error message from this line is :
AUTH
failed: ERR unknown command AUTH
, with args beginning with: ``,
the error message from this line is : AUTH failed: ERR unknown command AUTH, with args beginning with: ``,
@pangminfu, it looks like Predis is trying to connect to Sentinel with an empty password. Can you please post the stack trace for this exception?
We can also take a look at the Sentinel config to check that the application is loading the environment variables properly:
dd(app('config')->get('database.redis-sentinel'));
I also notice that laravel 5.1 is having issue with predis version 1.1 even before add in this package but i manage to solve it according to this solution below : nrk/predis#390 (comment)
So I was wonder could this cause "No sentinel server available for autodiscovery"? Also what is the value I should set for REDIS_DATABASE and what does this value mean?
This issue is not related to the problem you're seeing in this ticket. The REDIS_DATABASE
environment variable contains an integer that represents the Redis database number to use for a connection. A single Redis server can contain several databases (16 by default). We can use these to separate data as needed by a particular application. More info.
The client issues a SELECT
command to choose the database specified by REDIS_DATABASE
. This only happens after the client successfully contacts a Sentinel server to query it for information about the Redis master or replicas. If you don't need to use multiple databases, set REDIS_DATABASE
to 0
.
@cyrossignol , I manage to resolve
AUTH failed: ERR unknown command AUTH, with args beginning with: ``,
by upgrading my sentinel from 4.x to 5.0. because sentinel doesn't support AUTH command for version 4.x but the add back the support at sentinel 5.0.
But another issue came out on the same line but different message :
SELECT
failed: ERR unknown command SELECT
, with args beginning with: 0
probably is the problem with the sentinel setup
@cyrossignol ,
I checked predis package Factory.php :
protected function prepareConnection(NodeConnectionInterface $connection)
{
$parameters = $connection->getParameters();
if (isset($parameters->password)) {
$connection->addConnectCommand(
new RawCommand(array('AUTH', $parameters->password))
);
}
if (isset($parameters->database)) {
$connection->addConnectCommand(
new RawCommand(array('SELECT', $parameters->database))
);
}
}
above function will send a SELECT command to sentinel whenever $parameters->database
is set. so if my configuration doesn't set any value for REDIS_DATABASE or REDIS_SENTINEL_DATABASE the $parameters->database
empty?
I had tested comment out
$connection->addConnectCommand(
new RawCommand(array('SELECT', $parameters->database))
);
solve the issue.
@pangminfu, the REDIS_DATABASE
and REDIS_SENTINEL_DATABASE
variables are used to connect to Redis after the application contacts a Sentinel. Like with REDIS_PASSWORD
, this package should not set those connection parameters for the Sentinel clients, so I'm not sure why you're seeing AUTH
and SELECT
commands on Sentinel connections. It seems like the application might be trying to use the standard Redis API to connect to Sentinel. It will be helpful to see the information that I mentioned in my previous message so we can better understand what's happening.
@cyrossignol ,
Hi @cyrossignol ,
Issue resolved it was caused by predis v1.1. I had upgraded my predis to v1.1.1 all the issue is resolved for more information can refer to this PR364 of predis.
Thanks, @pangminfu -- I'll update the version number in the documentation so this is more clear. Typically, we can use a version constraint like "predis/predis": "^1.1"
in composer.json to avoid issues like this. For projects that follow semantic versioning, Composer will automatically install the latest bugfix release without jumping up to a new major release version. I'm glad you got it working!
Thanks @cyrossignol
Hi,
Using laravel 5.3, when the replication option is passed to the redis server it gives the above error. If I do not pass this value, I get the message "READONLY: You cannot write against a read only slave".
On the host file if I use the port 26379, it is unable to execute commands SET or GET command through that server.
Also, noticed in the test cases, there is a stub for configuration file that is using port 6379 but in the documentation it is mentioned to use the 26379, which one to use for.
Trying to troubleshoot this issue from past few hours but no avail.
Will appreciate any help.
Thanks