RedisLabs / redis-cluster-proxy

A proxy for Redis clusters.
GNU Affero General Public License v3.0
993 stars 132 forks source link

redis-cluster-proxy crashing #61

Open matrix-t opened 4 years ago

matrix-t commented 4 years ago

The version is the release 1.0-beta2. The crash happened when the master to which redis-cluster-proxy connected and used as primary got suddenly unavailable.

crash log: 2020-05-12T09:01:41.809020738Z stdout F === PROXY BUG REPORT START: Cut & paste starting from here === 2020-05-12T09:01:41.809037669Z stdout F [2020-05-12 09:01:41.808/0] Redis Cluster Proxy 0.9.102 crashed by signal: 11 2020-05-12T09:01:41.809054952Z stdout F [2020-05-12 09:01:41.808/0] Crashed running the instruction at: 0x408c84 2020-05-12T09:01:41.809076298Z stdout F [2020-05-12 09:01:41.808/0] Accessing address: 0x8 2020-05-12T09:01:41.809141611Z stdout F [2020-05-12 09:01:41.808/0] Handling crash on thread: 0 2020-05-12T09:01:41.809172261Z stdout F 2020-05-12T09:01:41.809186323Z stdout F 2020-05-12T09:01:41.809201293Z stdout F ------ STACK TRACE ------ 2020-05-12T09:01:41.809246189Z stdout F EIP: 2020-05-12T09:01:41.809285105Z stdout F redis-cluster-proxy(listEmpty+0x24)[0x408c84] 2020-05-12T09:01:41.809299386Z stdout F 2020-05-12T09:01:41.80931411Z stdout F Backtrace: 2020-05-12T09:01:41.80932926Z stdout F redis-cluster-proxy(logStackTrace+0x2d)[0x40dc4d] 2020-05-12T09:01:41.809344458Z stdout F redis-cluster-proxy(sigsegvHandler+0x17a)[0x40e25a] 2020-05-12T09:01:41.80936098Z stdout F /lib64/libpthread.so.0(+0x132d0)[0x7f82fb59b2d0] 2020-05-12T09:01:41.809377083Z stdout F redis-cluster-proxy(listEmpty+0x24)[0x408c84] 2020-05-12T09:01:41.809399569Z stdout F redis-cluster-proxy(listRelease+0x9)[0x408cd9] 2020-05-12T09:01:41.809493698Z stdout F redis-cluster-proxy(resetCluster+0x36)[0x40b2b6] 2020-05-12T09:01:41.809521301Z stdout F redis-cluster-proxy(updateCluster+0x1ca)[0x40cf3a] 2020-05-12T09:01:41.809538809Z stdout F redis-cluster-proxy(proxyCommand+0x141f)[0x4193bf] 2020-05-12T09:01:41.809556442Z stdout F redis-cluster-proxy(processRequest+0x2d5)[0x41b7d5] 2020-05-12T09:01:41.809575586Z stdout F redis-cluster-proxy(readQuery+0x1ea)[0x41c96a] 2020-05-12T09:01:41.809610288Z stdout F redis-cluster-proxy(aeProcessEvents+0x101)[0x4098e1] 2020-05-12T09:01:41.809631791Z stdout F redis-cluster-proxy(aeMain+0x2b)[0x409cdb] 2020-05-12T09:01:41.809701201Z stdout F redis-cluster-proxy[0x411b8c] 2020-05-12T09:01:41.809738579Z stdout F /lib64/libpthread.so.0(+0x84f9)[0x7f82fb5904f9] 2020-05-12T09:01:41.809756341Z stdout F /lib64/libc.so.6(clone+0x3f)[0x7f82fb2c3f2f] 2020-05-12T09:01:41.8097703Z stdout F 2020-05-12T09:01:41.809782737Z stdout F 2020-05-12T09:01:41.809796559Z stdout F ------ INFO OUTPUT ------ 2020-05-12T09:01:41.809846874Z stdout F # Proxy 2020-05-12T09:01:41.80988196Z stdout F proxy_version:0.9.102 2020-05-12T09:01:41.809896773Z stdout F proxy_git_sha1:00000000 2020-05-12T09:01:41.809910317Z stdout F proxy_git_dirty:0 2020-05-12T09:01:41.809927546Z stdout F proxy_git_branch: 2020-05-12T09:01:41.809947738Z stdout F os:Linux 4.15.0-99-generic x86_64 2020-05-12T09:01:41.809968636Z stdout F arch_bits:64 2020-05-12T09:01:41.810042283Z stdout F multiplexing_api:epoll 2020-05-12T09:01:41.810063659Z stdout F gcc_version:8.2.1 2020-05-12T09:01:41.810083387Z stdout F process_id:31 2020-05-12T09:01:41.810103715Z stdout F threads:8 2020-05-12T09:01:41.810122553Z stdout F tcp_port:7777 2020-05-12T09:01:41.810140832Z stdout F uptime_in_seconds:3780 2020-05-12T09:01:41.810176828Z stdout F uptime_in_days:0 2020-05-12T09:01:41.810192157Z stdout F config_file: 2020-05-12T09:01:41.810206659Z stdout F acl_user:default 2020-05-12T09:01:41.810219165Z stdout F 2020-05-12T09:01:41.810234292Z stdout F # Memory 2020-05-12T09:01:41.81024989Z stdout F used_memory:8189696 2020-05-12T09:01:41.810267234Z stdout F used_memory_human:7.81M 2020-05-12T09:01:41.810287134Z stdout F total_system_memory:33728958464 2020-05-12T09:01:41.810307505Z stdout F total_system_memory_human:31.41G 2020-05-12T09:01:41.81032145Z stdout F 2020-05-12T09:01:41.810338765Z stdout F # Clients 2020-05-12T09:01:41.810358186Z stdout F connected_clients:1 2020-05-12T09:01:41.810379253Z stdout F max_clients:10000 2020-05-12T09:01:41.810407401Z stdout F thread_0_clinets:1 2020-05-12T09:01:41.81042417Z stdout F thread_1_clinets:0 2020-05-12T09:01:41.810438885Z stdout F thread_2_clinets:0 2020-05-12T09:01:41.810452562Z stdout F thread_3_clinets:0 2020-05-12T09:01:41.810468214Z stdout F thread_4_clinets:0 2020-05-12T09:01:41.810492261Z stdout F thread_5_clinets:0 2020-05-12T09:01:41.810506579Z stdout F thread_6_clinets:0 2020-05-12T09:01:41.810521053Z stdout F thread_7_clinets:0 2020-05-12T09:01:41.810533371Z stdout F 2020-05-12T09:01:41.810547706Z stdout F # Cluster 2020-05-12T09:01:41.810563877Z stdout F address: 2020-05-12T09:01:41.810583278Z stdout F entry_node::0 2020-05-12T09:01:41.810610768Z stdout F 2020-05-12T09:01:41.810623832Z stdout F 2020-05-12T09:01:41.810639479Z stdout F ---- SIZEOF STRUCTS ---- 2020-05-12T09:01:41.810653969Z stdout F clientRequest: 184 2020-05-12T09:01:41.810668392Z stdout F client: 224 2020-05-12T09:01:41.810691151Z stdout F redisClusterConnection: 48 2020-05-12T09:01:41.810715281Z stdout F clusterNode: 112 2020-05-12T09:01:41.810731337Z stdout F redisCluster: 104 2020-05-12T09:01:41.810744585Z stdout F list: 48 2020-05-12T09:01:41.81075805Z stdout F listNode: 24 2020-05-12T09:01:41.810780879Z stdout F rax: 24 2020-05-12T09:01:41.810797111Z stdout F raxNode: 4 2020-05-12T09:01:41.810811536Z stdout F raxIterator: 480 2020-05-12T09:01:41.810824488Z stdout F aeEventLoop: 88 2020-05-12T09:01:41.810838539Z stdout F aeFileEvent: 32 2020-05-12T09:01:41.810852474Z stdout F aeTimeEvent: 64 2020-05-12T09:01:41.810867864Z stdout F 2020-05-12T09:01:41.810886717Z stdout F 2020-05-12T09:01:41.810905355Z stdout F ------ REGISTERS ------ 2020-05-12T09:01:41.810921757Z stdout F 2020-05-12T09:01:41.810941988Z stdout F RAX:0000000000000025 RBX:0000000000000000 2020-05-12T09:01:41.810960358Z stdout F RCX:0000000000000000 RDX:0000000002545550 2020-05-12T09:01:41.810994563Z stdout F RDI:00007f82f000bad0 RSI:00007f82fb57caa8 2020-05-12T09:01:41.811009414Z stdout F RBP:00007f82f00186ff RSP:00007f82fb1c6a80 2020-05-12T09:01:41.811025552Z stdout F R8 :00000000024e1d80 R9 :00007f82fb1c7700 2020-05-12T09:01:41.811039822Z stdout F R10:6e6972756769666e R11:0000000000000246 2020-05-12T09:01:41.81105407Z stdout F R12:0000000000000006 R13:00007f82f000bad0 2020-05-12T09:01:41.811071411Z stdout F R14:00007f82f0012763 R15:00007f82f0018d60 2020-05-12T09:01:41.811092855Z stdout F RIP:0000000000408c84 EFL:0000000000010206 2020-05-12T09:01:41.811109636Z stdout F CSGSFS:002b000000000033 2020-05-12T09:01:41.81112797Z stdout F (00007f82fb1c6a8f) -> 000000000041eaf8 2020-05-12T09:01:41.811147177Z stdout F (00007f82fb1c6a8e) -> 0000000000000000 2020-05-12T09:01:41.811239222Z stdout F (00007f82fb1c6a8d) -> 00007f8200000000 2020-05-12T09:01:41.811273634Z stdout F (00007f82fb1c6a8c) -> 0000000000000000 2020-05-12T09:01:41.811289104Z stdout F (00007f82fb1c6a8b) -> 00007f82f0015730 2020-05-12T09:01:41.811302989Z stdout F (00007f82fb1c6a8a) -> 00007f82000018eb 2020-05-12T09:01:41.811332337Z stdout F (00007f82fb1c6a89) -> 000000000040cf3a 2020-05-12T09:01:41.811350356Z stdout F (00007f82fb1c6a88) -> 00007f82f00157f0 2020-05-12T09:01:41.811364233Z stdout F (00007f82fb1c6a87) -> 000000000040b2b6 2020-05-12T09:01:41.811378488Z stdout F (00007f82fb1c6a86) -> 00000000024e39e0 2020-05-12T09:01:41.811427156Z stdout F (00007f82fb1c6a85) -> 0000000000408cd9 2020-05-12T09:01:41.811453613Z stdout F (00007f82fb1c6a84) -> 00000000024e39e0 2020-05-12T09:01:41.811468478Z stdout F (00007f82fb1c6a83) -> 0000000000000006 2020-05-12T09:01:41.81148219Z stdout F (00007f82fb1c6a82) -> 0000000000000000 2020-05-12T09:01:41.811496113Z stdout F (00007f82fb1c6a81) -> 00007f82f000bad0 2020-05-12T09:01:41.811509149Z stdout F (00007f82fb1c6a80) -> 00000000024e1d80 2020-05-12T09:01:41.811522359Z stdout F 2020-05-12T09:01:41.811537515Z stdout F 2020-05-12T09:01:41.811555272Z stdout F ------ DUMPING CODE AROUND EIP ------ 2020-05-12T09:01:41.811575011Z stdout F Symbol: listEmpty (base: 0x408c60) 2020-05-12T09:01:41.811594552Z stdout F Module: redis-cluster-proxy (base 0x400000) 2020-05-12T09:01:41.811613902Z stdout F $ xxd -r -p /tmp/dump.hex /tmp/dump.bin 2020-05-12T09:01:41.811631723Z stdout F $ objdump --adjust-vma=0x408c60 -D -b binary -m i386:x86-64 /tmp/dump.bin 2020-05-12T09:01:41.811649746Z stdout F ------ 2020-05-12T09:01:41.811668888Z stdout F dump of function (hexdump of 164 bytes): 2020-05-12T09:01:41.811701156Z stdout F 41554989fd415455534883ec08488b4728488b1f4885c0742f488d68ff0f1f00498b45184c8b63084885c07406488b7b10ffd04889df4883ed014c89e3e8deae01004883fdff75d849c745080000000049c745000000000049c74528000000004883c4085b5d415c415dc30f1f440000534889fbe887ffffff4889df5be99eae010066662e0f1f8400000000000f1f00554889f5534889fbbf180000004883ec08e86aad 2020-05-12T09:01:41.811753959Z stdout F Function at 0x423b80 is zfree 2020-05-12T09:01:41.811824796Z stdout F Function at 0x408c60 is listEmpty 2020-05-12T09:01:41.811838776Z stdout F 2020-05-12T09:01:41.81185604Z stdout F 2020-05-12T09:01:41.81187299Z stdout F === PROXY BUG REPORT END. Make sure to include from START to END. === 2020-05-12T09:01:41.811891451Z stdout F 2020-05-12T09:01:41.811912467Z stdout F Please report the crash by opening an issue on github: 2020-05-12T09:01:41.81193608Z stdout F 2020-05-12T09:01:41.811960627Z stdout F https://github.com/artix75/redis-cluster-proxy/issues 2020-05-12T09:01:41.811978477Z stdout F 2020-05-12T09:01:42.291611403Z stderr F /usr/local/bin/start_redis_proxy.sh: line 130: 31 Segmentation fault (core dumped) redis-cluster-proxy