search

Snapchat / KeyDB

A Multithreaded Fork of Redis
https://keydb.dev
BSD 3-Clause "New" or "Revised" License
11.34k stars 572 forks source link

[CRASH] Crashes when enable-async-commands was enabled in the cache use case (NOT flash). #751

Open CrazyTennisFan opened 10 months ago

CrazyTennisFan commented 10 months ago

Crash report

In the config file, when enable-async-commands yes, and we run the following 4 scripts, ./memtier_benchmark-12 -s 192.168.162.114 -p 6379 -t 10 -c 100 --test-time=36000 --command='HSET test123 key 1' --key-minimum=1 --key-maximum=3000

./memtier_benchmark-12 -s 192.168.162.114 -p 6379 -t 10 -c 100 --test-time=36000 --command='HGET test123 key' --key-minimum=1 --key-maximum=3000

./memtier_benchmark-12 -s 192.168.162.114 -p 6379 -t 10 -c 100 --test-time=36000 --command='del test123'

./memtier_benchmark-12 -s 192.168.162.114 -p 6379 -t 10 -c 100 --test-time=36000 --command='HGETALL test123 '

the server crashes or is stuck in the pure cache use case.

Crash #1: === KEYDB BUG REPORT START: Cut & paste starting from here === 1:18:M 23 Nov 2023 14:05:33.952 # === ASSERTION FAILED === 1:18:M 23 Nov 2023 14:05:33.952 # ==> dict.cpp:986 'iter->fingerprint == dictFingerprint(iter->d)' is not true

------ STACK TRACE ------

Backtrace: /opt/redis/redis-server 0.0.0.0:6379() [0x597804] /opt/redis/redis-server 0.0.0.0:6379(hashTypeReleaseIterator(hashTypeIterator)+0x29) [0x60e749] /opt/redis/redis-server 0.0.0.0:6379(genericHgetallCommand(client, int)+0x148) [0x611098] /opt/redis/redis-server 0.0.0.0:6379(call(client, int)+0xa1) [0x688631] /opt/redis/redis-server 0.0.0.0:6379(processCommand(client, int)+0x810) [0x689640] /opt/redis/redis-server 0.0.0.0:6379(processCommandAndResetClient(client, int)+0x65) [0x64c265] /opt/redis/redis-server 0.0.0.0:6379(processInputBuffer(client, bool, int)+0x1f4) [0x64e314] /opt/redis/redis-server 0.0.0.0:6379(readQueryFromClient(connection)+0x749) [0x64eb19] /opt/redis/redis-server 0.0.0.0:6379(connSocketEventHandler(aeEventLoop, int, void*, int)+0x11e) [0x68f52e] /opt/redis/redis-server 0.0.0.0:6379(ProcessEventCore+0xfc) [0x596d0c] /opt/redis/redis-server 0.0.0.0:6379(aeProcessEvents+0x205) [0x597a15]

Crash #2: === KEYDB BUG REPORT START: Cut & paste starting from here === 1:15:M 22 Nov 2023 06:51:52.081 # KeyDB 255.255.255 crashed by signal: 11, si_code: 1 1:15:M 22 Nov 2023 06:51:52.081 # Accessing address: (nil) 1:15:M 22 Nov 2023 06:51:52.081 # Crashed running the instruction at: 0x598a23

------ STACK TRACE ------ EIP: /opt/redis/redis-server 0.0.0.0:6379(dictFindWithPrev+0x63) [0x598a23]

Backtrace: /usr/lib64/libpthread.so.0(+0x13280) [0x7f09fc698280] /opt/redis/redis-server 0.0.0.0:6379(dictFindWithPrev+0x63) [0x598a23] /opt/redis/redis-server 0.0.0.0:6379(dictFind+0x2d) [0x598aed] /opt/redis/redis-server 0.0.0.0:6379(hashTypeGetFromHashTable(robj_roptr, char const)+0x39) [0x612259] /opt/redis/redis-server 0.0.0.0:6379() [0x6154ca] /opt/redis/redis-server 0.0.0.0:6379(hgetCommand(client)+0x84) [0x615794] /opt/redis/redis-server 0.0.0.0:6379(call(client, int)+0xa1) [0x67ca01] /opt/redis/redis-server 0.0.0.0:6379(processCommand(client, int)+0x810) [0x67da10] /opt/redis/redis-server 0.0.0.0:6379(processCommandAndResetClient(client, int)+0x65) [0x5e47a5] /opt/redis/redis-server 0.0.0.0:6379(processInputBuffer(client, bool, int)+0x1f4) [0x5e7344] /opt/redis/redis-server 0.0.0.0:6379(processClients()+0xc9) [0x5e8909] /opt/redis/redis-server 0.0.0.0:6379() [0x69be92] /opt/redis/redis-server 0.0.0.0:6379(beforeSleep(aeEventLoop*)+0x16e) [0x59bcde]

Aditional information test-leydb.conf.txt

  1. OS distribution and version
  2. Steps to reproduce (if any)
CrazyTennisFan commented 10 months ago

@JohnSully @MalavanEQAlpha

paulmchen commented 10 months ago

@JohnSully John, here are the async command issues I mentioned last Friday. It looks like a race condition in the async command when enable-async-commands is turned on. You can find more details in this ticket. I appreciate your help.