Suggestion on either using AOF or Save setting

[GoogleCodeExporter]

Hi, 

I am a newby with Redis and we have a current setup which is running on Centos 
5.6 and Redis version 2.4-R6. On our current setup, we have disabled "Save" and 
enabled "appendonly" and "appendfsync everysec". 

We have a monitoring checking the redis status and from to time we are alerts 
that Redis is not responding. As from digging the logs, I have checked that the 
time we are getting alerts on unresponsive redis is the time where we find the 
following on our logs:

===============================
[21955] 05 Sep 20:34:59 * Starting automatic rewriting of AOF on 100% growth
[21955] 05 Sep 20:34:59 * Background append only file rewriting started by pid 
9802
[9802] 05 Sep 20:37:12 * SYNC append only file rewrite performed
[21955] 05 Sep 20:37:12 * Background append only file rewriting terminated with 
success
[21955] 05 Sep 20:37:12 * Parent diff flushed into the new append log file with 
success (5031 bytes)
[21955] 05 Sep 20:37:12 * Append only file successfully rewritten.
[21955] 05 Sep 20:37:13 * The new append only file was selected for future 
appends.
===============================

As from doing some research, the issue we have is similar to the one described 
in this thread:

http://code.google.com/p/redis/issues/detail?id=602

I was not able to get the exact duration where our redis was unresponsive since 
this was happening randomly but I guess this is just for a few seconds. I would 
like to check if changing our setting from appendonly to "save" will help avoid 
this random blockage fro Redis. The server we are getting problem are mostly 
our Slave Redis servers. Below is the output of INFO from Redis. Thank in 
advanced for the expert advice. 

redis_version:2.3.9
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:4505
uptime_in_seconds:677131
uptime_in_days:7
lru_clock:1524475
used_cpu_sys:175.60
used_cpu_user:206.50
used_cpu_sys_children:33.76
used_cpu_user_children:19.48
connected_clients:2
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:1876587808
used_memory_human:1.75G
used_memory_rss:1922879488
used_memory_peak:1876695952
used_memory_peak_human:1.75G
mem_fragmentation_ratio:1.02
mem_allocator:jemalloc-2.2.1
loading:0
aof_enabled:1
changes_since_last_save:48944100
bgsave_in_progress:0
last_save_time:1314801859
bgrewriteaof_in_progress:0
total_connections_received:518784
total_commands_processed:53300973
expired_keys:160
evicted_keys:0
keyspace_hits:35339131
keyspace_misses:24440871
hash_max_zipmap_entries:512
hash_max_zipmap_value:64
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:406016
vm_enabled:0
role:slave
aof_current_size:1792451429
aof_base_size:1262572780
aof_pending_rewrite:0
master_host:
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
db0:keys=2047168,expires=0
db3:keys=640,expires=0

Thank you, 
Jimson

Original issue reported on code.google.com by jpenaflo...@gmail.com on 8 Sep 2011 at 11:01

[GoogleCodeExporter]

Hi, 

Just an added info, I have tried rotating the redis.log of one of our server 
and noticed a few seconds where we had a PING alert for Redis on the server. I 
am not sure if this is a normal scenario or should the logs also cause the 
blockage on Redis when it is Rotated. If rotating the logs will cause this type 
of scenario, please advice on alternative option we can rotate logs and avoid 
it from getting big. 

Thanks,
Jimson

Original comment by jpenaflo...@gmail.com on 8 Sep 2011 at 1:06

[GoogleCodeExporter]

This is a known issue with Redis AOF log rewriting. You linked issue 602, that 
is the correct one.

Your hang is caused by Linux atomically renaming files and it causing a large 
amount of disk IOs. Using the SAVE configuration option may help some, but 
forking a large process can also hang Redis, it still does an atomic rename 
(though the replaced file is smaller, so this isn't as big a problem as it is 
with AOF), and you only have your data persisted once every X minutes, instead 
of every second.

Original comment by josiah.c...@gmail.com on 9 Sep 2011 at 10:42

[GoogleCodeExporter]

Hi Josiah, 

Thank you so much for the information. I had also been doing some reading and 
they are pointing to same problem. We are also additional research for our 
setup to have a work around this issue. 

I would like to check if the log rotation should also have the same affect as 
to renaming a AOF file. I have tried doing a log rotation "redis.log" and I 
have noticed the same problem. Not really sure if this is a know problem since 
I have not encountered a thread having the same information. I am still doing 
additional research on this item. As of now the item I am thinking as a work 
around is to clear the log file after copying it. Some logs will be lost during 
the process but I was thinking this will avoid replacing the current redis.log. 
I would like to get your opinion if this approach is OK. Thank you in advance.

Regards,
Jimson

Original comment by jpenaflo...@gmail.com on 10 Sep 2011 at 1:07

[GoogleCodeExporter]

Hello Jimson, Log rewriting should not have the same problem AFAIK, but it is 
worth investigating. Are you sure that the delay you are seeing is related to 
log rewriting and not AOF rewriting instead? (or .rdb creation in background?).

Salvatore

Original comment by anti...@gmail.com on 12 Sep 2011 at 2:16