search

Snapchat / KeyDB

A Multithreaded Fork of Redis
https://keydb.dev
BSD 3-Clause "New" or "Revised" License
11.02k stars 564 forks source link

[CRASH] BGSave crashes #827

Open akn01001 opened 2 months ago

akn01001 commented 2 months ago

Crash report

Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.

------ STACK TRACE ------
EIP:
/lib/aarch64-linux-gnu/libc.so.6(pthread_cancel+0x24) [0xe14c33d3b864]

Backtrace:
linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xe14c349819d0]
/lib/aarch64-linux-gnu/libc.so.6(pthread_cancel+0x24) [0xe14c33d3b864]
keydb-rdb-bgsave 10.101.1.20:6379(bioKillThreads+0x74) [0xadfaa71d25d8]
keydb-rdb-bgsave 10.101.1.20:6379(doFastMemoryTest()+0x38) [0xadfaa71b25a8]
keydb-rdb-bgsave 10.101.1.20:6379(_serverAssert+0x144) [0xadfaa71b2754]
keydb-rdb-bgsave 10.101.1.20:6379(rdbSaveRio(_rio*, redisDbPersistentDataSnapshot const**, int*, int, rdbSaveInfo*)+0x4b4) [0xadfaa717bab4]
keydb-rdb-bgsave 10.101.1.20:6379(rdbSaveFile(char*, redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x128) [0xadfaa717cc3c]
keydb-rdb-bgsave 10.101.1.20:6379(rdbSave(redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x64) [0xadfaa717cf44]
keydb-rdb-bgsave 10.101.1.20:6379(rdbSaveBackgroundFork(rdbSaveInfo*)+0x124) [0xadfaa717d614]
keydb-rdb-bgsave 10.101.1.20:6379(rdbSaveBackground(rdbSaveInfo*)+0xa8) [0xadfaa717da68]
keydb-rdb-bgsave 10.101.1.20:6379(serverCron(aeEventLoop*, long long, void*)+0xc78) [0xadfaa71247fc]
keydb-rdb-bgsave 10.101.1.20:6379(aeProcessEvents+0x264) [0xadfaa711f534]
keydb-rdb-bgsave 10.101.1.20:6379(aeMain+0xb4) [0xadfaa711f9a4]
keydb-rdb-bgsave 10.101.1.20:6379(workerThreadMain(void*)+0xc0) [0xadfaa7138e20]
/lib/aarch64-linux-gnu/libc.so.6(+0x7d5c8) [0xe14c33d3d5c8]
/lib/aarch64-linux-gnu/libc.so.6(+0xe5edc) [0xe14c33da5edc]

------ REGISTERS ------
14500:675:C 30 Apr 2024 22:46:42.599 #
X18:0000adfaa7986268 X19:0000e14c311fd0e0
X20:0000000000000001 X21:0000e14c2b3fd0e0
X22:0000adfaa774d000 X23:0000adfaa77620d0
X24:0000adfaa7605538 X25:0000000000000001
X26:000000000018a873 X27:0000adfaa774f000
X28:0000adfaa7755b78 X29:0000e14c2b3facb0
X30:0000adfaa71d25d8
pc:0000e14c33d3b864 sp:0000e14c2b3facb0
pstate:0000000020001000 fault_address:0000e14c311fd1b0

14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facbf) -> 0000e14c1c39755b
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facbe) -> 0000adfaa7200f44
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facbd) -> 0000e14c2b3fad30
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facbc) -> c0e56993eeab8300
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facbb) -> 0000adfaa77620d0
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facba) -> 0000adfaa71b1fb0
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb9) -> 0000e14c2b3fadb0
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb8) -> 0000adfaa75f8420
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb7) -> 0000000400000005
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb6) -> 0000000600000007
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb5) -> 000000000000055c
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb4) -> 0000adfaa75f8420
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb3) -> 0000000000000001
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb2) -> 0000adfaa777af18
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb1) -> 0000adfaa71d25d8
14500:675:C 30 Apr 2024 22:46:42.599 # (0000e14c2b3facb0) -> 0000e14c2b3fad80

Aditional information

  1. OS distribution and version - Ubuntu 22.04 on aws arm grav3 m7.2xl 32gb memory, 15gb dataset
  2. Steps to reproduce (if any)
keithchew commented 2 months ago

Looking at your stack trace, it is interesting it does not print out the assert error, but proceeds further to the memory test. You did not indicate the version you are using, but just in case it is v6.3.4, you might want to check out:

https://github.com/Snapchat/KeyDB/issues/743

akn01001 commented 2 months ago

Yes, i am using 6.3.4. The link you mentions says to use 6.3.2?

keithchew commented 2 months ago

It mentions not to use v6.3.4 until it is fixed, so if you confirm that it is the same problem you are having, you can try to downgrade to a version that works for you.