tporadowski / redis

Native port of Redis for Windows. Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs. This repository contains unofficial port of Redis to Windows.
http://redis.io
Other
8.88k stars 1.09k forks source link

Redis crashing every couple of hours: EXCEPTION_ACCESS_VIOLATION #128

Closed downey-lv closed 2 years ago

downey-lv commented 2 years ago

Describe the bug/issue The instance is crashing every couple of hours while not performing anything intensive. Restarting the service brings it back

To Reproduce The instance is serving HGETALL throughout the day and is updated once per 24h. Crashes every couple of hours. RAM is free, pagination settings are fine, test-memory passed. No special config, max memory is set to 2G, database is ~700MB

Redis version=6.2.6 OS Name Microsoft Windows Server 2019 Standard Version 10.0.17763 Build 17763

Crash report

=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 6.2.6
[10580] 01 Feb 19:11:25.668 # --- EXCEPTION_ACCESS_VIOLATION
[10580] 01 Feb 19:11:25.668 # --- STACK TRACE
redis-server.exe!((null):0)(0x1401B77F0, 0x025AFF60, 0x025AE8A0, 0x025AE8A0)
redis-server.exe!((null):0)(0x025AE8A0, 0x025AFF01, 0x1400FCE50, 0x140109EB0)
KERNELBASE.dll!UnhandledExceptionFilter((null):0)(0x025AEEC0, 0x7FFB35242020, 0x00000000, 0x00000000)
ntdll.dll!memset((null):0)(0x00000000, 0x00000000, 0x025AF530, 0x025AEE68)
ntdll.dll!_C_specific_handler((null):0)(0x00000000, 0x025AEE50, 0x00000000, 0x140000000)
ntdll.dll!_chkstk((null):0)(0x025AEE50, 0x00000000, 0x7FFB3273CC3C, 0x7FFB32690000)
ntdll.dll!RtlWalkFrameChain((null):0)(0x00000000, 0x025AF098, 0x14020A080, 0x025AF0B0)
ntdll.dll!KiUserExceptionDispatcher((null):0)(0x00000000, 0x14010DAD1, 0x00000000, 0x029B0720)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x025AF918, 0x00000008, 0x00000000, 0x029B0730)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00000006, 0x00000000, 0x00000002, 0x7FCD00000000)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00000002, 0xFFFFFFFF0000001B, 0x00000000, 0x186A000000000)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00002954, 0x1401B1610, 0x00171080, 0x00008000)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x004B79D0, 0x004B79D0, 0x0049FD60, 0x004B79D0)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x01653CE0, 0x01653CE0, 0x01653CE0, 0x004B79D0)
redis-server.exe!KiUserExceptionDispatcher((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart((null):0)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[10580] 01 Feb 19:11:25.696 # --- INFO OUTPUT
# Server
redis_version:6.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:164b73bbc4ccd2e9
redis_mode:standalone
os:Windows  
arch_bits:64
multiplexing_api:WinSock_IOCP
atomicvar_api:pthread-mutex
gcc_version:0.0.0
process_id:10580
process_supervised:no
run_id:37438219c8cf954fca2f7de9ab06679e645e98ee
tcp_port:6379
server_time_usec:1643735485663412
uptime_in_seconds:5398
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:16345533
executable:C:\redis\"c:\redis\redis-server.exe"
config_file:C:\redis\redis.conf
io_threads_active:0

# Clients
connected_clients:65
cluster_connections:0
maxclients:3168
client_recent_max_input_buffer:8
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:747906048
used_memory_human:713.26M
used_memory_rss:747904256
used_memory_rss_human:713.26M
used_memory_peak:747906048
used_memory_peak_human:713.26M
used_memory_peak_perc:100.00%
used_memory_overhead:322993304
used_memory_startup:393568
used_memory_dataset:424912744
used_memory_dataset_perc:56.84%
allocator_allocated:748082072
allocator_active:1216348160
allocator_resident:1233125376
total_system_memory:0
total_system_memory_human:0B
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:2147483648
maxmemory_human:2.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.63
allocator_frag_bytes:468266088
allocator_rss_ratio:1.01
allocator_rss_bytes:16777216
rss_overhead_ratio:0.61
rss_overhead_bytes:-485221120
mem_fragmentation_ratio:1.00
mem_fragmentation_bytes:0
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:1331720
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1-redis
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1643730087
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:3351
total_commands_processed:21444
instantaneous_ops_per_sec:0
total_net_input_bytes:874114
total_net_output_bytes:14170206
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:2
evicted_keys:0
keyspace_hits:8396
keyspace_misses:1
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:10
dump_payload_sanitizations:0
total_reads_processed:19587
total_writes_processed:19279
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:07e983cd39ff89fe20f736d6f05cc1bbe577b7d5
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:2.781250
used_cpu_user:10.062500
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

# Modules

# Commandstats
cmdstat_info:calls=3248,usec=386313,usec_per_call=118.94,rejected_calls=0,failed_calls=0
cmdstat_client:calls=3248,usec=9815,usec_per_call=3.02,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=3286,usec=20952,usec_per_call=6.38,rejected_calls=0,failed_calls=0
cmdstat_role:calls=3248,usec=20114,usec_per_call=6.19,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=8397,usec=35294,usec_per_call=4.20,rejected_calls=9,failed_calls=0
cmdstat_ping:calls=17,usec=0,usec_per_call=0.00,rejected_calls=1,failed_calls=0

# Errorstats
errorstat_LOADING:count=10

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=6353978,expires=0,avg_ttl=0
[10580] 01 Feb 19:11:25.698 # 
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Event Log Don't see anything in particular here, redis fork operation was completed successfully 2 hours before the crash.

Additional context

[10580] 01 Feb 17:41:27.161 # Redis is starting ......
[10580] 01 Feb 17:41:27.161 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=10580, just started
[10580] 01 Feb 17:41:27.162 # Configuration loaded
[10580] 01 Feb 17:41:27.165 * Running mode=standalone, port=6379.
[10580] 01 Feb 17:41:27.165 # Server initialized
[10580] 01 Feb 17:41:27.166 * Loading RDB produced by version 6.2.6
[10580] 01 Feb 17:41:27.166 * RDB age 7705 seconds
[10580] 01 Feb 17:41:27.166 * RDB memory usage when created 0.00 Mb
[10580] 01 Feb 17:41:33.772 # RDB file was saved with checksum disabled: no check performed.
[10580] 01 Feb 17:41:33.773 # Done loading RDB, keys loaded: 6353978, keys expired: 0.
[10580] 01 Feb 17:41:33.773 * DB loaded from disk: 6.608 seconds
[10580] 01 Feb 17:41:33.773 * Ready to accept connections
tporadowski commented 2 years ago

It seems this is a version created based on this fork, but available here: https://github.com/zkteco-home/redis-windows @zkteco-home any comments?

downey-lv commented 2 years ago

My bad, I posted it where the app said to, didn't realize this wasn't the proper fork. I'll bring it over to his github and probably switch to another (yours) version for now.