Closed michael-stevens closed 6 years ago
trinitum
commented
6 years ago "No callbacks in the queue" means that you've got unexpected reply from the server. "You can't use RedisDB::execute when you have replies to fetch" means that you are trying to run synchronous command before fetching reply to the previously submitted asynchronous one. I can't tell exactly what's wrong without seeing the code. Are you using threads?
michael-stevens
commented
6 years ago Hi,
We aren't using threads, I can replicate the problem with a single threaded command line script (although I haven't quite managed to minimise it to the point it only uses RedisDB yet).
The problem is most easily triggered by restarting the redis instance that RedisDB is connected to, although we've also seen it occur under load.
I'm still trying to confirm this but it looks like the problem occurs if the first recv after connecting and setting the timeouts fails with ECONNRESET. Wondering if RedisDB doesn't reset its state properly if the connection fails during setup? Speculation though.
trinitum
commented
6 years ago script would be great
michael-stevens
commented
6 years ago Hi,
Okay this is still a bit hacky, but this script:
#!/usr/bin/perl -w
use strict;
use RedisDB;
while (1) {
my $redis = RedisDB->new( host => '192.168.204.29', port => '6379', raise_error => 0, timeout => 0.000001);
sleep 1;
print $redis->exists('scripts:query_cache');
print "\n";
}Plus modifyingRedisDB.pm to simulate a network problem, by changing _recv_data_nb:
my $ret = recv( $self->{_socket}, my $buf, 131072, $DONTWAIT );
$ret = undef;
$! = ECONNRESET;seem to consistently cause a problem for me.
(a problem beyond the fact you'd obviously expect the call to fail with a network error)
trinitum
commented
6 years ago Silly me trying to support too many ways to use this module... I believe the following patch should fix the problem, I'll release the new version later tonight:
diff --git a/lib/RedisDB.pm b/lib/RedisDB.pm
index 00a9a57..b2958d8 100644
--- a/lib/RedisDB.pm
+++ b/lib/RedisDB.pm
@@ -445,9 +445,11 @@ sub _recv_data_nb {
last if $! == EAGAIN or $! == EWOULDBLOCK;
next if $! == EINTR;
- # on any other error close connection
- $self->_on_disconnect( 1,
- RedisDB::Error::DISCONNECTED->new("Error reading from server: $!") );
+ # on any other error close the connection
+ my $error =
+ RedisDB::Error::DISCONNECTED->new("Error reading from server: $!");
+ $self->_on_disconnect( 1, $error );
+ return $error;
}
elsif ( $buf ne '' ) {I was getting a different error though: Can't use an undefined value as a symbol reference at lib/RedisDB.pm line 441., did it gave to you "No callbacks"?
michael-stevens
commented
6 years ago I was getting the symbol reference error with that test case. but I'm hoping it's the same underlying bug.
While debugging I was getting a bit worried that you don't reset the connection state in on_disconnect if raise_error is not set:
if ($err) {
$error_obj ||= RedisDB::Error::DISCONNECTED->new(
"Server unexpectedly closed connection. Some data might have been lost.");
if ( $self->{raise_error} or $self->{_in_multi} or $self->{_watching} ) {
$self->reset_connection;
die $error_obj;
}But I don't understand the code well enough to be sure if it's a problem.
We'll test your patch, many thanks.
trinitum
commented
6 years ago reset_connection is not a very good name, it resets the whole object to the initial state, but with raise_error disabled it is not the solution, so it calls propagate_error instead to deliver the error to all the callbacks and reply queue, and removes the socket (which effectively resets network connection).
michael-stevens
commented
6 years ago Testing the patch, it's looking good so far.
trinitum
commented
6 years ago v2.54 is on CPAN. Thank you for reporting the problem and investigating.
Hi,
We're getting a series of
Error in Perl code: No callbacks in the queue and no default callback set at /usr/local/share/perl5/RedisDB.pmerrors.
We've also seen:
Error in Perl code: You can't use RedisDB::execute when you have replies to fetch..This was a previously working application and it's not clear what's triggering the problem, although it's possibly related to an upgrade to RHEL 7.4.
perl is:
and redis is:
RedisDB is 2.53.