Snapchat / KeyDB

A Multithreaded Fork of Redis
https://keydb.dev
BSD 3-Clause "New" or "Revised" License
11.53k stars 578 forks source link

[CRASH] nil address on clientsCronResizeQueryBuffer on replica #877

Open keithchew opened 1 month ago

keithchew commented 1 month ago

Testing on async_flash branch, but looks like this bug is present before that.

Crash happens on replica/slave:

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
7:101:S 16 Oct 2024 08:19:33.512 # KeyDB 6.3.4 crashed by signal: 11, si_code: 128
7:101:S 16 Oct 2024 08:19:33.512 # Accessing address: (nil)
7:101:S 16 Oct 2024 08:19:33.512 # Crashed running the instruction at: 0x555c13c76f44

------ STACK TRACE ------
EIP:
/opt/KeyDB/bin/keydb-server *:6379(sdsAllocSize+0x4) [0x555c13c76f44]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fc7f34d8420]
/opt/KeyDB/bin/keydb-server *:6379(sdsAllocSize+0x4) [0x555c13c76f44]
/opt/KeyDB/bin/keydb-server *:6379(clientsCronResizeQueryBuffer(client*)+0x70) [0x555c13c61cd0]
/opt/KeyDB/bin/keydb-server *:6379(clientsCron(int)+0x184) [0x555c13c62384]
/opt/KeyDB/bin/keydb-server *:6379(serverCron(aeEventLoop*, long long, void*)+0x229) [0x555c13c632b9]
/opt/KeyDB/bin/keydb-server *:6379(aeProcessEvents+0x235) [0x555c13c5a135]
/opt/KeyDB/bin/keydb-server *:6379(aeMain+0x3e) [0x555c13c5aaee]
/opt/KeyDB/bin/keydb-server *:6379(workerThreadMain(void*)+0x12b) [0x555c13c7441b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fc7f34cc609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fc7f33ef133]

Not sure if this is only applicable for a replica...

keithchew commented 1 month ago

It looks like this has also been reported upstream:

https://github.com/redis/redis/issues/13137

Some discussions about this here, but not sure if it fixes the issue: https://github.com/redis/redis/pull/11766 https://github.com/redis/redis/pull/12924

I will try to pull those changes on my local and retest...