RedisLabs / redis-cluster-proxy

A proxy for Redis clusters.
GNU Affero General Public License v3.0
990 stars 129 forks source link

Redis Cluster Proxy Crash When I write redis data in batches #114

Open JackDan9 opened 1 year ago

JackDan9 commented 1 year ago

[2023-03-03 11:46:20.643/M] Redis Cluster Proxy v999.999.999 (unstable) [2023-03-03 11:46:20.643/M] Commit: (ac83840d/1) [2023-03-03 11:46:20.643/M] Git Branch: unstable [2023-03-03 11:46:20.643/M] PID: 1 [2023-03-03 11:46:20.643/M] OS: Linux 3.10.0-957.el7.x86_64 x86_64 [2023-03-03 11:46:20.643/M] Bits: 64 [2023-03-03 11:46:20.643/M] Log level: error [2023-03-03 11:46:20.643/M] Connections pool size: 10 (respawn 2 every 50ms if below 10) [2023-03-03 11:46:20.644/M] Starting 8 threads... [2023-03-03 11:46:20.644/M] Fetching cluster configuration... [2023-03-03 11:46:20.647/M] Cluster Address: redis-cluster:6379 [2023-03-03 11:46:20.647/M] Cluster has 3 masters and 3 replica(s) [2023-03-03 11:46:20.731/M] All thread(s) started! [2023-03-03 11:49:55.003/6] Thread 0 terminated [2023-03-03 11:49:55.003/6] Thread 1 terminated [2023-03-03 11:49:55.003/6] Thread 2 terminated [2023-03-03 11:49:55.003/6] Thread 3 terminated [2023-03-03 11:49:55.003/6] Thread 4 terminated [2023-03-03 11:49:55.003/6] Thread 5 terminated [2023-03-03 11:49:55.003/6] Thread 7 terminated

=== PROXY BUG REPORT START: Cut & paste starting from here === [2023-03-03 11:49:55.003/6] Redis Cluster Proxy 999.999.999 crashed by signal: 11 [2023-03-03 11:49:55.003/6] Crashed running the instruction at: 0x40831c [2023-03-03 11:49:55.003/6] Accessing address: 0x1c0 [2023-03-03 11:49:55.003/6] Handling crash on thread: 6

------ STACK TRACE ------ EIP: redis-cluster-proxy(listDelNode+0xc)[0x40831c]

Backtrace: redis-cluster-proxy(logStackTrace+0x2d)[0x40d0fd] redis-cluster-proxy(sigsegvHandler+0x186)[0x40d716] /lib64/libpthread.so.0(+0xf630)[0x7f1538239630] redis-cluster-proxy(listDelNode+0xc)[0x40831c] redis-cluster-proxy[0x410d2f] redis-cluster-proxy[0x419d88] redis-cluster-proxy(aeProcessEvents+0x13b)[0x408d7b] redis-cluster-proxy(aeMain+0x2b)[0x40913b] redis-cluster-proxy[0x410f9c] /lib64/libpthread.so.0(+0x7ea5)[0x7f1538231ea5] /lib64/libc.so.6(clone+0x6d)[0x7f1537f5a96d]

------ INFO OUTPUT ------

Proxy

proxy_version:999.999.999 proxy_git_sha1:ac83840d proxy_git_dirty:1 proxy_git_branch:unstable os:Linux 3.10.0-957.el7.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:7.3.1 process_id:1 threads:8 tcp_port:7777 uptime_in_seconds:215 uptime_in_days:0 config_file:/data/proxy.conf acl_user:default

Memory

used_memory:17357512 used_memory_human:16.55M total_system_memory:405186953216 total_system_memory_human:377.36G

Clients

connected_clients:6 max_clients:10000 thread_0_clinets:1 thread_1_clinets:1 thread_2_clinets:1 thread_3_clinets:0 thread_4_clinets:1 thread_5_clinets:1 thread_6_clinets:1 thread_7_clinets:0

Cluster

address: entry_node::0

---- SIZEOF STRUCTS ---- clientRequest: 184 client: 224 redisClusterConnection: 48 clusterNode: 112 redisCluster: 104 list: 48 listNode: 24 rax: 24 raxNode: 4 raxIterator: 480 aeEventLoop: 88 aeFileEvent: 32 aeTimeEvent: 64

------ REGISTERS ------

RAX:000000000109ae70 RBX:00000000000001c0 RCX:00007f15382237b8 RDX:0000000000000006 RDI:000000000109aeb0 RSI:00000000000001c0 RBP:000000000109aeb0 RSP:00007f15347cddd0 R8 :0000000000000046 R9 :000000000042be54 R10:0000000000000004 R11:373736310a0d3031 R12:0000000000000000 R13:000000000109aeb0 R14:000000000108adc0 R15:00000000010acfa0 RIP:000000000040831c EFL:0000000000010202 CSGSFS:0000000000000033 (00007f15347cdddf) -> 00007f1518261570 (00007f15347cddde) -> 0000000000000001 (00007f15347cdddd) -> 0000000000000143 (00007f15347cdddc) -> 00000000010acfa4 (00007f15347cdddb) -> 0000000000000143 (00007f15347cddda) -> 000000000118d3f0 (00007f15347cddd9) -> 000000061825ee60 (00007f15347cddd8) -> 0000000000000000 (00007f15347cddd7) -> 0000000000419ea8 (00007f15347cddd6) -> 000000000042b45d (00007f15347cddd5) -> 0000000000419d88 (00007f15347cddd4) -> 00007f151818e6a0 (00007f15347cddd3) -> 0000000000410d2f (00007f15347cddd2) -> 00007f1518261570 (00007f15347cddd1) -> 00007f151818e6a0 (00007f15347cddd0) -> 00007f1518261570

------ DUMPING CODE AROUND EIP ------ Symbol: listDelNode (base: 0x408310) Module: redis-cluster-proxy (base 0x400000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x408310 -D -b binary -m i386:x86-64 /tmp/dump.bin

dump of function (hexdump of 140 bytes): 55534889fd4889f34883ec08488b16488b46084885d2743848894208488b46084885c07433488910488b45184885c07406488b7b10ffd04889dfe801ad010048836d28014883c4085b5dc30f1f4400004885c048890775cd48895508ebca669055534889fdbf1000000089f34883ec08e8bbab01004885c0740e85db7412488b55084889108958084883c408 Function at 0x423050 is zfree Function at 0x422f40 is zmalloc

=== PROXY BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       https://github.com/artix75/redis-cluster-proxy/issues