search

microsoftarchive / redis

Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes
http://redis.io
Other
20.81k stars 5.37k forks source link

Redis crashes on Windows Server 2012 #496

Open Vinno97 opened 8 years ago

Vinno97 commented 8 years ago

I have been noticing Redis crashes that can only be solved by restarting the VM.

We have Redis 2.8.21 running on a Windows Server 2012 VM with Node.JS and MySQL and use it for sessions, pub/sub and "message" queues.

I have ran redis --test-memory, but cut it off (ctrl+c) after about 7-10 hours when I was not sure if it would go on forever (does it?). I did not notice any anomalies and it was at Random fill (41)

The log file is almost 3GB and mainly contains two messages: The first one is about 95% of the file and contains only these three lines:

[1356] 18 Aug 11:23:55.804 # Can't save in background: fork: Invalid argument
[1356] 18 Aug 11:23:55.914 * 1 changes in 900 seconds. Saving...
[1356] 18 Aug 11:23:55.914 # BeginForkOperation: system error caught. error code=0x000005af, message=BeginForkOperation: VirtualProtect failed: unknown error

These got spammed every second until this happened:

=== REDIS BUG REPORT START: Cut & paste starting from here ===

[1136] 20 Aug 12:27:43.147 # --- EXCEPTION_ACCESS_VIOLATION
[1136] 20 Aug 12:27:43.147 # --- INFO OUTPUT
# Server
redis_version:2.8.21
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:e84b20ff4e0813fa
redis_mode:standalone
os:Windows  
arch_bits:64
multiplexing_api:winsock_IOCP
process_id:1136
run_id:a1c83fe426c9bf71c7b552d5dee7031ef423ca1e
tcp_port:6379
uptime_in_seconds:94279
uptime_in_days:1
hz:10
lru_clock:12071071
config_file:C:\Program Files\Redis\redis.windows-service.conf

# Clients
connected_clients:1575
client_longest_output_list:808529968
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:50062176
used_memory_human:47.74M
used_memory_rss:50061224
used_memory_peak:50094648
used_memory_peak_human:47.77M
used_memory_lua:36864
mem_fragmentation_ratio:1.00
mem_allocator:dlmalloc-2.8

# Persistence
loading:0
rdb_changes_since_last_save:1460
rdb_bgsave_in_progress:0
rdb_last_save_time:1471688800
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:1471688801
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

# Stats
total_connections_received:1575
total_commands_processed:4470967
instantaneous_ops_per_sec:43
total_net_input_bytes:297207540
total_net_output_bytes:0
instantaneous_input_kbps:2.78
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:44806
evicted_keys:0
keyspace_hits:4408216
keyspace_misses:70
pubsub_channels:743
pubsub_patterns:0
latest_fork_usec:218190

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:370.14
used_cpu_user:174.39
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Commandstats
cmdstat_get:calls=2345603,usec=20979874,usec_per_call=8.94
cmdstat_setex:calls=54757,usec=764566,usec_per_call=13.96
cmdstat_del:calls=38,usec=227,usec_per_call=5.97
cmdstat_select:calls=2,usec=6,usec_per_call=3.00
cmdstat_expire:calls=2062683,usec=18896605,usec_per_call=9.16
cmdstat_auth:calls=1575,usec=6001,usec_per_call=3.81
cmdstat_info:calls=1575,usec=574890,usec_per_call=365.01
cmdstat_subscribe:calls=1573,usec=15769,usec_per_call=10.02
cmdstat_unsubscribe:calls=830,usec=15783,usec_per_call=19.02
cmdstat_publish:calls=2331,usec=38394,usec_per_call=16.47

# Keyspace
db1:keys=42593,expires=42593,avg_ttl=33511210
[1136] 20 Aug 12:27:43.162 # 
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

At this moment the Node.JS server kept crashing because it could not access it's sessions anymore. (It sort of looks like it only happens when it's trying to save a session, but I could be wrong)

P.S. I have tried to be smart and configure the service to restart the computer on the third crash, but that does not seem to work.

P.P.S Redis 3.2 is currently running on our test server and we'll be upgrading to this soon.