yst726 / redis

Automatically exported from code.google.com/p/redis
0 stars 0 forks source link

temp rdb files get left behind and copy on write fails #438

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago
What steps will reproduce the problem?
1. Running an ec2 instance with 36GB of RAM 
2.  Large amount of keys being inserted and causing the dumping to disk to hang 
and cause io wait to spike up and make the server unresponsive.  Temp-*.rdb 
files get left behind during this and the dump.rdb file is never copied over

What is the expected output? What do you see instead?
temp-*.rdb files should not be left behind and during the dump to disk 

What version of the product are you using? On what operating system?
2.2 RC3 , CentOS 5.5 64bit

Please provide any additional information below.
daemonize no
pidfile /var/run/redis.pid
port 6379
timeout 300
loglevel notice
logfile /log/redis/redis.log
databases 16
save 3600 100000
rdbcompression no
dbfilename dump.rdb
dir /data
maxmemory 32212254720
appendonly no
 appendfsync no
vm-enabled no
vm-swap-file /tmp/redis.swap
vm-max-memory 0
vm-page-size 32
vm-pages 134217728
vm-max-threads 4
glueoutputbuf yes
hash-max-zipmap-entries 64
hash-max-zipmap-value 512
activerehashing yes
maxmemory-policy allkeys-lru

I've tried adjusting the save time from 5 minutes to up to an hour and seeing 
cases where the copy on write process seems to overlap and ends up leaving 
temp-*.rdb files behind and not replacing the dump.rdb file with the new temp 
dump.  Here's the information from the redis-cli

redis> info
redis_version:2.1.11
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:20460
uptime_in_seconds:472
uptime_in_days:0
lru_clock:1629928
used_cpu_sys:102.71
used_cpu_user:19.58
used_cpu_sys_childrens:0.00
used_cpu_user_childrens:0.00
connected_clients:21
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:17372111968
used_memory_human:16.18G
used_memory_rss:22643032064
mem_fragmentation_ratio:1.30
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:11
bgsave_in_progress:0
last_save_time:1295561529
bgrewriteaof_in_progress:0
total_connections_received:1838
total_commands_processed:75490
expired_keys:31
evicted_keys:0
keyspace_hits:51333
keyspace_misses:12892
hash_max_zipmap_entries:64
hash_max_zipmap_value:512
pubsub_channels:1
pubsub_patterns:0
vm_enabled:0
role:master
allocation_stats:6=1,8=10090,9=2172,10=1924,11=849,12=67098,13=41812,14=386252,1
5=24011001,16=163041667,17=156,18=1162,19=1201,20=3672,21=81126,22=130924266,23=
34990475,24=154613856,25=365,26=817,27=3705,28=45627,29=224084,30=8740912,31=253
,32=2241,33=1198,34=4669,35=33058,36=129234,37=5536858,38=198,39=273,40=874,41=3
602,42=24811,43=93525,44=2246114,45=568,46=903,47=1748,48=11860,49=23182,50=7417
4,51=2381866,52=251,53=426,54=1355,55=4903,56=18350,57=59215,58=1052540,59=899,6
0=1363,61=2528,62=4978,63=19659,64=55273,65=1380000,66=373,67=501,68=1050,69=275
6,70=14767,71=44966,72=642340,73=1313,74=1855,75=3000,76=5814,77=17891,78=47613,
79=939978,80=365,81=537,82=1106,83=2809,84=12864,85=38017,86=446094,87=1745,88=4
368,89=3772,90=6449,91=17529,92=44504,93=695695,94=464,95=623,96=1239,97=2987,98
=11775,99=33928,100=330903,101=2222,102=3036,103=4497,104=7501,105=17771,106=422
56,107=542944,108=554,109=774,110=1373,111=3165,112=11344,113=30977,114=258018,1
15=2710,116=3670,117=5430,118=8783,119=18340,120=39489,121=437103,122=673,123=94
0,124=1577,125=3414,126=10842,127=28366,128=207126,129=3277,130=4505,131=6310,13
2=10684,133=18706,134=37645,135=363715,136=799,137=1148,138=1822,139=4887,140=12
834,141=26347,142=168132,143=3884,144=5323,145=7073,146=10675,147=19435,148=3512
1,149=303546,150=1072,151=1371,152=2014,153=3953,154=10622,155=24473,156=140395,
157=4783,158=5983,159=7925,160=10842,161=18247,162=32289,163=257534,164=1272,165
=1604,166=2466,167=4292,168=10432,169=22940,170=117680,171=5256,172=6463,173=813
9,174=11093,175=17438,176=29175,177=221099,178=1503,179=1885,180=2853,181=4860,1
82=10316,183=21719,184=100505,185=5839,186=6865,187=8277,188=10695,189=15848,190
=25836,191=192094,192=1824,193=2263,194=3160,195=5101,196=10282,197=20523,198=86
626,199=6287,200=7161,201=8078,202=9900,203=14193,204=22396,205=166274,206=2228,
207=2703,208=3570,209=5452,210=10207,211=19496,212=74809,213=6430,214=6767,215=7
625,216=8997,217=12458,218=19573,219=146328,220=2577,221=2939,222=4058,223=5849,
224=10211,225=18654,226=65629,227=6203,228=6344,229=6774,230=7859,231=11059,232=
17206,233=130537,234=2760,235=3342,236=4309,237=5923,238=9829,239=17155,240=5763
3,241=5670,242=5767,243=6031,244=6672,245=9719,246=15558,247=116994,248=3131,249
=3665,250=4575,251=6132,252=9735,253=16095,254=50638,255=5134,>=256=4883013
db0:keys=12142561,expires=12142560
db2:keys=65163180,expires=65163180

Original issue reported on code.google.com by jli...@gmail.com on 20 Jan 2011 at 10:23

GoogleCodeExporter commented 8 years ago
Same problem here, redis-2.8.13, Ubuntu Utopic.

Moreover, I'm running two redis servers with different settings. The temp files 
names don't contain any component that would help me figure out which of those 
two servers leave these behind.

Original comment by egm...@gmail.com on 13 Mar 2015 at 4:05

GoogleCodeExporter commented 8 years ago
This Google Code project is outdated. Please report new bugs to 
https://github.com/antirez/redis/issues/

Original comment by jeredi...@gmail.com on 13 Mar 2015 at 4:12