redis locking

[GoogleCodeExporter]

What version of Redis you are using, in what kind of Operating System?
2.2 

What is the problem you are experiencing?
No, just wants to know if redis will lock the db  while writing or reading from 
it? I have a query which reads around 1 lack records from redis, but i noticed 
same time redis getting stuck and even "info" command not working . Is it 
normal?

What steps will reproduce the problem?

Do you have an INFO output? Please past it here.

redis_version:2.2.0
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:24829
uptime_in_seconds:249837
uptime_in_days:2
lru_clock:142324
used_cpu_sys:9589.27
used_cpu_user:2645.72
used_cpu_sys_childrens:731.97
used_cpu_user_childrens:134.06
connected_clients:1
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:2235368128
used_memory_human:2.08G
used_memory_rss:3167801344
mem_fragmentation_ratio:1.42
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:2241
bgsave_in_progress:0
last_save_time:1301657468
bgrewriteaof_in_progress:0
total_connections_received:13739616
total_commands_processed:89080005
expired_keys:0
evicted_keys:0
keyspace_hits:70052629
keyspace_misses:5922576
hash_max_zipmap_entries:64
hash_max_zipmap_value:512
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0
role:master

If it is a crash, can you please paste the stack trace that you can find in
the log file or on standard output? This is really useful for us!

Please provide any additional information below.

-Vij

Original issue reported on code.google.com by vijeeshk...@gmail.com on 1 Apr 2011 at 6:38

[GoogleCodeExporter]

Hi,

Redis is single threaded by design. It never locks the database,
but it cannot perform queries in parallel: all of them are
serialized.

Regards,
Didier.

Original comment by didier...@gmail.com on 1 Apr 2011 at 7:42

[GoogleCodeExporter]

That means multiple queries cant be be at same time?

-Vij

Original comment by vijeeshk...@gmail.com on 1 Apr 2011 at 8:02

[GoogleCodeExporter]

Yes but redis is so fast you won't notice. Install redis and run 
redis-benchmark to get an idea of how fast it is.

Original comment by rowan%wo...@gtempaccount.com on 1 Apr 2011 at 8:30

[GoogleCodeExporter]

Thanks. I have some code which does sunion which is very heavy, whenever i run 
this,server seems blocking other operations (i can see lot of open connections 
in netstat , which i think was created by other php scripts on my webserver , 
which runs simple redis operations which cant close the connection as the redis 
query is pending ) . Is that a normal behaviour?

-Vij

Original comment by vijeeshk...@gmail.com on 1 Apr 2011 at 8:36

[GoogleCodeExporter]

How big are your sets? How many sets are you performing a union on? How big is 
the resulting set?

Original comment by josiah.c...@gmail.com on 1 Apr 2011 at 9:43

[GoogleCodeExporter]

hmm..30 sets with 1 million data each..so we are doing sunion on 30m data

-Vij

Original comment by vijeeshk...@gmail.com on 1 Apr 2011 at 10:57

[GoogleCodeExporter]

That could take a minute or so depending on the speed of the machine Redis is 
running on. You may want to consider running two copies of Redis. One to handle 
all of your small operations, and a slave that handles those large union 
operations.

That is, the behavior of Redis isn't a bug.

To me, it sounds like you are doing some sort of 30-day unique calculation. If 
so, there are almost-as-fast methods of doing that calculation, but without 
stalling Redis. We can discuss it on the list if you are interested.

Original comment by josiah.c...@gmail.com on 1 Apr 2011 at 11:16

[GoogleCodeExporter]

Thanks.
Yes we are doing 30 days unique calculation. Each set has 300,000 values and we 
check 
unique keys for 30 days records. Now this calculation takes few secs. But it 
blocks other operations. Can you please let me know what all are the methods we 
have to do this without stalling Redis? Running copies of redis, do you  mean 
redis master - slave servers?

-Vij

Original comment by vijeeshk...@gmail.com on 3 Apr 2011 at 12:08

[GoogleCodeExporter]

Using a slave to perform the calculation would work just fine.

Alternatively, rather than storing the data as a set, you could store them as a 
zset all with the same score. Why? When two items in a zset have the same 
score, they are are ordered based on the lexicographic order of the member. 
That means that the sets are sorted by the item you want to access. Once you 
have sorted data, you can iterate over all of your sets in parallel to discover 
your unique entries. Here's an implementation in Python that should work:

import heapq

def yield_zset_members(conn, key, blocksize=1000):
    start = 0
    chunk = [None]
    while chunk:
        chunk = conn.zrange(key, start, start+blocksize-1)
        start += len(chunk)
        for item in chunk:
            yield item

def find_unique(conn, keys):
    last = None
    unique = 0
    for item in heapq.merge(*[yield_zset_members(conn, key) for key in keys]):
        if item != last:
            unique += 1
        last = item
    return unique

You can use "ZUNIONSTORE key 1 key WEIGHTS 1" on each key one at a time to 
initially convert your sets into zsets. You should only need to do this once if 
you switch your code to use zsets instead of sets. They will take a bit more 
memory to store, each "zadd" operation will take slightly more time than the 
comparable "sadd" operation (O(logn) vs. O(1)), and the overall time to 
complete the "how many unique items are there?" will be higher, but Redis won't 
pause noticeably during these operations (except for the initial set -> zset 
conversion).

One small note if you do go with this zset-based version, make sure that none 
of your zsets are changing during the operation, or your results could be 
*very* wrong.

Original comment by josiah.c...@gmail.com on 4 Apr 2011 at 7:28

[GoogleCodeExporter]

Thanks. We are moving that query to slave 
i got one more issue regarding this, i tried to connect to redis when the issue 
happened, in netstat i can see around 150 connections are open. I tried to 
connect using a php script with 5 secs timeout set . But whenever the 
established connections reach around ~150 script started failing to connect. it 
doesnt look   normal behaviour,as the server has no maxclients limit.   Can you 
please advice

-Vij 

Original comment by vijeeshk...@gmail.com on 5 Apr 2011 at 7:28

[GoogleCodeExporter]

Is this happening when Redis is stalling due to a large operation? Are you 
using your own hardware (that you can verify good), or are you using a provider 
(like Amazon, Rackspace, etc.)? We've had strange network connectivity issues 
on occasion with Amazon, and replacing the instance fixes it. Check top to see 
if maybe your system is busy doing something else (forking during bgsave, etc.).

You may want to try to explicitly set the maxclients to something like 1000, 
500, or whatever your platform limits processes to by default. Also, sometimes 
processes will get default limits that are lower than the platform limit. Look 
into adjusting those upwards.

Original comment by josiah.c...@gmail.com on 5 Apr 2011 at 7:56