zrangebyscore returning inconsistent results

[GoogleCodeExporter]

using redis 2.0.3 on ubuntu 9.10

We see occasionally a very weird behavior of zrangebyscore the problem is 
zrangebyscore can't see that some elements are present.
Here I pasted a ruby 'irb' session where I've seen the problem happening
I was the only one client connected on that server at the moment this was 
happening.

the short story is:

1) the set has cardinality 1
2) I execute few zrangebyscore to "find" the key
3) The key doesn't appear to be anywhere
4) zrangebyscore with '-inf', '+inf' and  with_scores somehow magically reveals 
the key (at score 1 which should havbe been picked up by all the previous 
queries)
5) now repeating the initial zrangebyscore command is possible to see teh key 
again.

>> $redis.zcard mykey
=> 1
>> $redis.zrangebyscore mykey, 0, 1000
=> []
>> $redis.zrangebyscore mykey, -1000, 1000
=> []
>> $redis.zrangebyscore mykey, -1000, 1000_000_000_000
=> []
>> $redis.zrangebyscore mykey, -1000_000_000_000, 1000_000_000_000
=> []
>> $redis.zrangebyscore mykey, '-inf', '+inf'
=> ["m:saw:1"]
>> $redis.zrangebyscore mykey, '-inf', '+inf', :with_scores => true
=> ["m:saw:1", "1"]
>> $redis.zrangebyscore mykey, '1', '1'
=> ["m:saw:1"]
>> $redis.zrangebyscore mykey, '0', '1000'
=> ["m:saw:1"]

Original issue reported on code.google.com by hungrybl...@gmail.com on 4 Nov 2010 at 4:43

[GoogleCodeExporter]

Hello, I guess this happens when this single-element sorted set was created 
after a lot of hacking against a sorted set, or are you able to provide the 
list of commands that can reproduce this?

It can help a lot to know the kind of commands you run against sorted sets, the 
kind of deletion you use (example by range) and so forth.

Thanks,
Salvatore

Original comment by anti...@gmail.com on 4 Nov 2010 at 4:50

[GoogleCodeExporter]

Forgot to ask a few things:

1) Does this happen consistently over time?
2) Did you tried if with redis-cli this is also the case?

Thanks,
Salvatore

Original comment by anti...@gmail.com on 4 Nov 2010 at 4:51

[GoogleCodeExporter]

No it doesn't happen every time.
It happens under conditions I'm still trying to chase.
We have a fairly busy system ~ 20000 commands / sec where this occurs about 
twice a day
But once it starts happening the zrangebyscore is broken and seems to be fixed 
only by calling -inf +inf
I didn't try with the redis-cli but I'll do so the next time it happens.

What I suspect is that this is triggered by a broken call i.e. with a very 
weird score to zadd with that same value.

In this example there's only one key in the zset but it also happens in zsets 
with multiple values, in these cases only one key become invisible while others 
keep working correctly.

Original comment by hungrybl...@gmail.com on 4 Nov 2010 at 5:10

[GoogleCodeExporter]

I forgot to add that entries are never removed from this set.
Also, in theory entries should never have their score changed(but I suspect 
under some very rare conditions our system is actually trying to zadd the same 
key with a non properly formatted score)

Original comment by hungrybl...@gmail.com on 4 Nov 2010 at 5:15

[GoogleCodeExporter]

This is a truly strange issue actually... what is more shocking for me is that 
a -inf +inf range fetching fixes things. It's a read only operation that does 
not change the sorted set...

If you can try with redis-cli we can make sure it's not a matter if redis-rb 
parsing error.
Another thing that could help a lot is if you can enable AOF in some instance, 
so we could reach this point just playing the AOF, with some luck.

Thank you very much,
it is very unlikely that we can fix this issue in short time without your help 
:)

Salvatore

Original comment by anti...@gmail.com on 4 Nov 2010 at 5:32

[GoogleCodeExporter]

I'll keep the ticket updated with my findings when this will happen again.

At the moment I'm updating the redis-rb version from 2.0.6 to 2.0.13  on all 
our servers so it might also be I won't see the issue again (2.0.12 includes a 
fix for a bug where redis-rb after a connection timeout mangles the data 
passing over the network).

Enabling AOF is not an option since out of 20000 commands each second about 50% 
are writes.

Original comment by hungrybl...@gmail.com on 4 Nov 2010 at 5:45

[GoogleCodeExporter]

Thank you for your help!

Salvatore

Original comment by anti...@gmail.com on 4 Nov 2010 at 5:49