redis terminates unexpectedly when adding members to a sorted set

[GoogleCodeExporter]

What version of Redis you are using, in what kind of Operating System?
I am using redis 2.2.4 under a 64-bit Ubuntu 8.04.4 LTS. Specifically, it is a 
high memory extra large instance in EC2. redis is built 32-bit.

What is the problem you are experiencing?
I have a program that parses words from the stored keys in redis and creates an 
index of the words. For example, if the word "foo" can be found in the value of 
the redis key "bar", then the entry "bar" would be added to the sorted set 
"word:foo". The score used is a Unix timestamp. The program uses the redis-py 
client version 2.2.4.

When the program has processed about 518,000-519,000 keys, redis unexpectedly 
terminates, and the redis-py client exits with an error 
"redis.exceptions.ConnectionError: Error 111 connecting localhost:6382. 
Connection refused."

What steps will reproduce the problem?
1. Get the value of a key from redis.
2. Parse the value for words.
3. Get the metaphone keys of the retrieved words.
4. Add the name of the key to the sorted set corresponding to the metaphone 
keys of the retrieved words.
5. Repeat 518,000-519,000 times.

Do you have an INFO output? Please past it here.
I have attached a .tar.gz containing the redis configuration used, the redis 
log file, and the INFO output.

If it is a crash, can you please paste the stack trace that you can find in
the log file or on standard output? This is really useful for us!
No stack trace was produced.

Please provide any additional information below.

Original issue reported on code.google.com by luip...@gmail.com on 19 Apr 2011 at 11:46

Attachments:

• redis_info.tar.gz

[GoogleCodeExporter]

Hello, is it possible that this is due just to the Linux OOM killer killing the 
process for out of memory, pelase can you verify this in your dmesg output? 
Thank you.

Original comment by anti...@gmail.com on 19 Apr 2011 at 11:50

[GoogleCodeExporter]

Thanks for replying.

I can't see any reference to redis in the output of dmesg. I have attached it 
here for your analysis.

Original comment by luip...@gmail.com on 19 Apr 2011 at 12:04

Attachments:

• dmesg_output.gz

[GoogleCodeExporter]

What do you see in the log or standard output of the terminal running Redis? I 
suspect that Redis is unable to allocate more memory since it is compiled as 
32-bit. This would not trigger OOM, but rather return a NULL when allocating 
new memory which causes Redis to abort. Can you retry against an instance that 
is compiled for a 64-bit target? If you have enough memory and the same doesn't 
happen around ~500K keys, we'll know that a 32-bit compiled Redis cannot 
address enough memory to hold the data you're putting in...

Original comment by pcnoordh...@gmail.com on 19 Apr 2011 at 1:29

[GoogleCodeExporter]

Thank you for the suggestion. That seems to be the cause as my program was now 
able to run successfully.

Is this an inherent limitation of 32-bit programs?

Original comment by luip...@gmail.com on 20 Apr 2011 at 2:50

[GoogleCodeExporter]

Yes, when a program is compiled for a 32-bit target its addressing space is 
limited to 32 bits, or 4GB. Good to hear this solved your issue.

Original comment by pcnoordh...@gmail.com on 20 Apr 2011 at 7:56

• Changed state: Invalid