search

Snapchat / KeyDB

A Multithreaded Fork of Redis
https://keydb.dev
BSD 3-Clause "New" or "Revised" License
11.02k stars 564 forks source link

[BUG] Crash calling the KEYS command #826

Open keithchew opened 2 months ago

keithchew commented 2 months ago

Testing on v6.3.4 (async_flash branch, although I don't think it is related to FLASH).

I have added a cleanup client to delete keys while other clients are still reading. Encountered this crash:

7:63:M 24 Apr 2024 13:15:21.612 # === ASSERTION FAILED ===
7:63:M 24 Apr 2024 13:15:21.612 # ==> networking.cpp:300 'c->conn == nullptr || c->lock.fOwnLock()' is not true

------ STACK TRACE ------

Backtrace:
/opt/KeyDB/bin/keydb-server *:6379(prepareClientToWrite(client*)+0x139) [0x5614af7a8179]
/opt/KeyDB/bin/keydb-server *:6379(addReplyProto(client*, char const*, unsigned long)+0x17) [0x5614af7a82f7]
/opt/KeyDB/bin/keydb-server *:6379(keysCommandCore(client*, redisDbPersistentDataSnapshot const*, char*)+0x175) [0x5614af7bf705]
/opt/KeyDB/bin/keydb-server *:6379(+0x26db62) [0x5614af7c9b62]
/opt/KeyDB/bin/keydb-server *:6379(AsyncWorkQueue::WorkerThreadMain()+0x450) [0x5614af8b5f30]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f10bf6fddf4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f10bf4a2609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f10bf3c5353]

A bit lost on how to track down the root cause, but will keep testing and post any updates here.

keithchew commented 2 weeks ago

This happened again, and looking at the stack trace again, it is crashing on the "keys" command. There are 2 code flows which can cause this, will add some more logging to track this down.