Crash when using BLPOP and MGET

[doyoubi]

Version

unstable branch commit 6751bf515fcef0a46c273f0199e49794592529ec

Steps to Reproduce

> redis-cli -p 7777
127.0.0.1:7777> PROXY CONFIG SET enable-cross-slot 1
OK
127.0.0.1:7777> blpop a b 0
(error) ERR Cross-slot queries are not supported for this command
127.0.0.1:7777> mget a b
^C
Wait 5 seconds and type Ctrl-C to stop the redis-client could sometimes trigger proxy crash.

Crash Log:

=== PROXY BUG REPORT START: Cut & paste starting from here ===
[2020-03-24 15:16:55.273/0] Redis Cluster Proxy 999.999.999 crashed by signal: 11
[2020-03-24 15:16:55.273/0] Crashed running the instruction at: 0x10fbd05c5
[2020-03-24 15:16:55.273/0] Accessing address: 0x0
[2020-03-24 15:16:55.273/0] Handling crash on thread: 0

------ STACK TRACE ------
EIP:
0   redis-cluster-proxy                 0x000000010fbd05c5 listNext + 21

Backtrace:
0   redis-cluster-proxy                 0x000000010fbd5fc2 logStackTrace + 114
1   redis-cluster-proxy                 0x000000010fbd63c1 sigsegvHandler + 577
2   libsystem_platform.dylib            0x00007fff74483b5d _sigtramp + 29
3   ???                                 0x000000000000ffff 0x0 + 65535
4   redis-cluster-proxy                 0x000000010fbe1618 freeClient + 472
5   redis-cluster-proxy                 0x000000010fbd1328 aeProcessEvents + 744
6   redis-cluster-proxy                 0x000000010fbd166b aeMain + 43
7   redis-cluster-proxy                 0x000000010fbe7914 execProxyThread + 52
8   libsystem_pthread.dylib             0x00007fff7448c2eb _pthread_body + 126
9   libsystem_pthread.dylib             0x00007fff7448f249 _pthread_start + 66
10  libsystem_pthread.dylib             0x00007fff7448b40d thread_start + 13

------ INFO OUTPUT ------
# Proxy
proxy_version:999.999.999
proxy_git_sha1:6751bf51
proxy_git_dirty:0
proxy_git_branch:unstable
os:Darwin 18.7.0 x86_64
arch_bits:64
multiplexing_api:kqueue
gcc_version:4.2.1
process_id:98168
threads:1
tcp_port:7777
uptime_in_seconds:10
uptime_in_days:0
config_file:
acl_user:default

# Memory
used_memory:899664
used_memory_human:878.58K
total_system_memory:8589934592
total_system_memory_human:8.00G

# Clients
connected_clients:0
max_clients:10000

# Cluster
address:
entry_node::0

---- SIZEOF STRUCTS ----
clientRequest: 184
client: 224
redisClusterConnection: 48
clusterNode: 112
redisCluster: 104
list: 48
listNode: 24
rax: 24
raxNode: 4
raxIterator: 480
aeEventLoop: 88
aeFileEvent: 32
aeTimeEvent: 64

------ REGISTERS ------

RAX:d000000000000000 RBX:00007fa5f0f01d20
RCX:0000000000000001 RDX:00000000000f66d0
RDI:000070000280fe20 RSI:00007fa5f0c00000
RBP:000070000280fe10 RSP:000070000280fe10
R8 :0000000000000003 R9 :0000000000000000
R10:0000000000000004 R11:0000000000000004
R12:00007fa5f0d00e60 R13:00007fa5f0f00c6b
R14:0000000000000001 R15:000070000280fe20
RIP:000000010fbd05c5 EFL:0000000000010246
CS :000000000000002b FS:0000000000000000  GS:0000000000000000
(000070000280fe1f) -> 0000000b00000001
(000070000280fe1e) -> 0000000000000014
(000070000280fe1d) -> 0000000000000001
(000070000280fe1c) -> 00007fa5f0f00270
(000070000280fe1b) -> 000000010fbd1328
(000070000280fe1a) -> 000070000280fed0
(000070000280fe19) -> 0000000000000000
(000070000280fe18) -> 0000000000000001
(000070000280fe17) -> 00007fa5f0f00270
(000070000280fe16) -> 0000000000000001
(000070000280fe15) -> 000000010fc4c000
(000070000280fe14) -> 00007fa5f0f00270
(000070000280fe13) -> 0000000000000000
(000070000280fe12) -> d000000000000000
(000070000280fe11) -> 000000010fbe1618
(000070000280fe10) -> 000070000280fe60

------ DUMPING CODE AROUND EIP ------
Symbol: listNext (base: 0x10fbd05b0)
Module: /xxxxxx/redis-cluster-proxy/./src/redis-cluster-proxy (base 0x10fbcf000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x10fbd05b0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
dump of function  (hexdump of 149 bytes):
554889e5488b074885c0741031c9837f08000f94c1488b0cc848890f5dc36690554889e5415741564154534889fbbf30000000e838ea01004885c00f84840100004989c648c740280000000048c740200000000048c740180000000048c740100000000048c740080000000048c70000000000f30f6f4310f30f7f4010488b4320498946204c8b3b4d85ff0f843701000066480f7e
Function at 0x10fbef020 is zmalloc

=== PROXY BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           https://github.com/artix75/redis-cluster-proxy/issues

[1]    98168 segmentation fault  ./src/redis-cluster-proxy --threads 1 127.0.0.1:8899

[artix75]

@doyoubi Thank you, I will investigate on this.

[artix75]

@doyoubi I cannot reproduce the issue, I've tried your steps many times but always without problems.

[doyoubi]

The OS is Mac OSX. The backend Redis cluster is just a single node with all the slots:

127.0.0.1:8899> cluster nodes
eb1adebbd80b6d6ce7c6a5e37f4f040b2dbe138b :8899@18899 myself,master - 0 0 0 connected 0-16383

The last MGET will block. Then press ctrl-c to trigger crash.

Does the traceback help?

* thread #2, stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00000001000015c5 redis-cluster-proxy`listNext(iter=0x000070000d1ade20) at adlist.c:0
    frame #1: 0x0000000100012618 redis-cluster-proxy`freeClient [inlined] freeAllClientRequests(c=<unavailable>) at proxy.c:2737:18
    frame #2: 0x0000000100012524 redis-cluster-proxy`freeClient(c=0x0000000100501790) at proxy.c:2780
    frame #3: 0x0000000100002328 redis-cluster-proxy`aeProcessEvents(eventLoop=0x00000001004018f0, flags=11) at ae.c:451:17
    frame #4: 0x000000010000266b redis-cluster-proxy`aeMain(eventLoop=0x00000001004018f0) at ae.c:521:9
    frame #5: 0x0000000100018914 redis-cluster-proxy`execProxyThread(ptr=0x00000001004011a0) at proxy.c:4771:5
    frame #6: 0x00007fff7448c2eb libsystem_pthread.dylib`_pthread_body + 126
    frame #7: 0x00007fff7448f249 libsystem_pthread.dylib`_pthread_start + 66
    frame #8: 0x00007fff7448b40d libsystem_pthread.dylib`thread_start + 13

[artix75]

@doyoubi Ok, I've been able to replicate it. Now I'm working to fix it. Thank you very much!

[artix75]

@doyoubi Fixed in 9a2383d