search

RedisLabs / redis-cluster-proxy

A proxy for Redis clusters.
GNU Affero General Public License v3.0
990 stars 129 forks source link

Crash from `proxy cluster update` #66

Open dacjames opened 4 years ago

dacjames commented 4 years ago

To reproduce:

  1. Start 6 node cluster: 3 master, 3 slave.
  2. Create the cluster using redis-cli --cluster create. Default options.
  3. Start redis-cluster-proxy. Default options.
  4. Connect to proxy via redis-cli.
  5. Run the following series of commands:
    • proxy cluster info
    • proxy cluster update
    • proxy cluster info
    • proxy cluster update.

This will crash the proxy, with the following error message:

redis-cluster-proxy(77503,0x700009d67000) malloc: *** error for object 0x7fa03fa0f920: pointer being freed was not allocated
redis-cluster-proxy(77503,0x700009d67000) malloc: *** set a breakpoint in malloc_error_break to debug

Note that the second proxy cluster info command does not return the expected results.

If you start fresh again and run the following in sequence:

You get a different, but seemingly related crash. The first crash does not produce a bug report, but here is the report for the second sequence:

=== PROXY BUG REPORT START: Cut & paste starting from here ===
[2020-06-02 14:18:48.539/0] Redis Cluster Proxy 999.999.999 crashed by signal: 11
[2020-06-02 14:18:48.539/0] Crashed running the instruction at: 0x10476f0b3
[2020-06-02 14:18:48.539/0] Accessing address: 0x0
[2020-06-02 14:18:48.539/0] Handling crash on thread: 0

------ STACK TRACE ------
EIP:
0   redis-cluster-proxy                 0x000000010476f0b3 listRelease + 35

Backtrace:
0   redis-cluster-proxy                 0x0000000104774c22 logStackTrace + 114
1   redis-cluster-proxy                 0x000000010477501f sigsegvHandler + 575
2   libsystem_platform.dylib            0x00007fff6ee215fd _sigtramp + 29
3   ???                                 0x0000000000000000 0x0 + 0
4   redis-cluster-proxy                 0x0000000104772320 resetCluster + 64
5   redis-cluster-proxy                 0x0000000104773a99 updateCluster + 697
6   redis-cluster-proxy                 0x000000010477c81a proxyCommand + 4122
7   redis-cluster-proxy                 0x0000000104780e5d processRequest + 989
8   redis-cluster-proxy                 0x0000000104782c3e readQuery + 494
9   redis-cluster-proxy                 0x00000001047700d8 aeProcessEvents + 728
10  redis-cluster-proxy                 0x000000010477041b aeMain + 43
11  redis-cluster-proxy                 0x00000001047865d4 execProxyThread + 52
12  libsystem_pthread.dylib             0x00007fff6ee2d109 _pthread_start + 148
13  libsystem_pthread.dylib             0x00007fff6ee28b8b thread_start + 15

------ INFO OUTPUT ------
# Proxy
proxy_version:999.999.999
proxy_git_sha1:ac83840d
proxy_git_dirty:0
proxy_git_branch:unstable
os:Darwin 19.4.0 x86_64
arch_bits:64
multiplexing_api:kqueue
gcc_version:4.2.1
process_id:77962
threads:8
tcp_port:7777
uptime_in_seconds:9
uptime_in_days:0
config_file:./proxy.conf
acl_user:default

# Memory
used_memory:9540368
used_memory_human:9.10M
total_system_memory:17179869184
total_system_memory_human:16.00G

# Clients
connected_clients:1
max_clients:10000
thread_0_clinets:1
thread_1_clinets:0
thread_2_clinets:0
thread_3_clinets:0
thread_4_clinets:0
thread_5_clinets:0
thread_6_clinets:0
thread_7_clinets:0

# Cluster
address:
entry_node::0

---- SIZEOF STRUCTS ----
clientRequest: 184
client: 224
redisClusterConnection: 48
clusterNode: 112
redisCluster: 104
list: 48
listNode: 24
rax: 24
raxNode: 4
raxIterator: 480
aeEventLoop: 88
aeFileEvent: 32
aeTimeEvent: 64

------ REGISTERS ------

RAX:0000000000000b03 RBX:3532313366012828
RCX:0000000000000000 RDX:00000000000ecbb0
RDI:00007fee3040f060 RSI:00007fee32900000
RBP:0000700001b72ae0 RSP:0000700001b72ac0
R8 :0000000000000005 R9 :0000000000000001
R10:00007fee32900000 R11:00007fee329043e0
R12:00007fee32913340 R13:0000000000000006
R14:00007fee3040f060 R15:0000000000636432
RIP:000000010476f0b3 EFL:0000000000010202
CS :000000000000002b FS:0000000000000000  GS:0000000000000000
(0000700001b72acf) -> 0000000000000000
(0000700001b72ace) -> 0000000000000000
(0000700001b72acd) -> 0000000000000000
(0000700001b72acc) -> 0000000000000000
(0000700001b72acb) -> 0000000104773a99
(0000700001b72aca) -> 0000700001b72d70
(0000700001b72ac9) -> 00007fee329043e3
(0000700001b72ac8) -> 0000000000000000
(0000700001b72ac7) -> 0000000000000006
(0000700001b72ac6) -> 00007fee32913340
(0000700001b72ac5) -> 0000000104772320
(0000700001b72ac4) -> 0000700001b72b10
(0000700001b72ac3) -> 00007fee30706bf0
(0000700001b72ac2) -> 00007fee30706bf0
(0000700001b72ac1) -> 00007fee32913340
(0000700001b72ac0) -> 0000000000000000

------ DUMPING CODE AROUND EIP ------
Symbol: listRelease (base: 0x10476f090)
Module: /usr/local/bin/redis-cluster-proxy (base 0x10476e000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x10476f090 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
dump of function  (hexdump of 163 bytes):
554889e5415741564154534989fe4c8b7f284d85ff742f498b1e660f1f44000049ffcf4c8b6308498b46184885c07406488b7b10ffd04889dfe8f2ea01004c89e34d85ff75da49c746280000000049c746080000000049c706000000004c89f75b415c415e415f5de9c3ea01000f1f00554889e54156534989f64889fbbf18000000e889e901004885c074234c897010488b4b284885c9741a48c70000000000488b13
Function at 0x10478dbc0 is zfree
Function at 0x10478daa0 is zmalloc

=== PROXY BUG REPORT END. Make sure to include from START to END. ===