search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
729 stars 1.4k forks source link

Docker-database's redis-server's redis6 instance crashed #19227

Open sg893052 opened 4 months ago

sg893052 commented 4 months ago

Description

Docker-database's redis-server (version 6.2.6)'s COUNTERS_DB(redis6) instance crashed

Steps to reproduce the issue:

  1. Longevity test

Describe the results you received:

May 23 21:22:38.293225+00:00 2024 leaf5 CRIT systemd-coredump[2285207]: Process 1253 (redis-server) of user 0 dumped core.#012#012Stack trace of thread 75:#012#0 0x00007f5a6dd37087 kill (libc.so.6 + 0x39087)#012#1 0x000055e587d5ffe5 n/a (/usr/bin/redis-check-rdb + 0xa5fe5)

May 23 21:22:37.232176+00:00 2024 leaf5 INFO database#supervisord: redis6 75:M 23 May 2024 21:22:37.231 # Redis 6.2.6 crashed by signal: 11, si_code: 1 May 23 21:22:37.232600+00:00 2024 leaf5 INFO database#supervisord: redis6 75:M 23 May 2024 21:22:37.232 # Accessing address: 0xffffffff830ff539 May 23 21:22:37.232749+00:00 2024 leaf5 INFO database#supervisord: redis6 75:M 23 May 2024 21:22:37.232 # Crashed running the instruction at: 0x55e587d110f9 May 23 21:22:37.234150+00:00 2024 leaf5 INFO database#supervisord: redis6 May 23 21:22:37.234150+00:00 2024 leaf5 INFO database#supervisord: redis6 ------ STACK TRACE ------ May 23 21:22:37.234545+00:00 2024 leaf5 INFO database#supervisord: redis6 EIP: May 23 21:22:37.234895+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(sdsMakeRoomFor+0x429)[0x55e587d110f9] May 23 21:22:37.235203+00:00 2024 leaf5 INFO database#supervisord: redis6 May 23 21:22:37.235203+00:00 2024 leaf5 INFO database#supervisord: redis6 Backtrace: May 23 21:22:37.235482+00:00 2024 leaf5 INFO database#supervisord: redis6 /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f5a6dee7140] May 23 21:22:37.235792+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(sdsMakeRoomFor+0x429)[0x55e587d110f9] May 23 21:22:37.236126+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(sdscatlen+0x44)[0x55e587d11794] May 23 21:22:37.236469+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(notifyKeyspaceEvent+0xd6)[0x55e587d89fe6] May 23 21:22:37.236787+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(hsetCommand+0xf9)[0x55e587d51819] May 23 21:22:37.237001+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(call+0xb1)[0x55e587d09bb1] May 23 21:22:37.237323+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(luaRedisGenericCommand+0x3b9)[0x55e587d76df9] May 23 21:22:37.237749+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(+0x117858)[0x55e587dd1858] May 23 21:22:37.237993+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(+0x122020)[0x55e587ddc020] May 23 21:22:37.238379+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(+0x117ef5)[0x55e587dd1ef5] May 23 21:22:37.238674+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(+0x1171eb)[0x55e587dd11eb] May 23 21:22:37.238994+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(+0x1180a0)[0x55e587dd20a0] May 23 21:22:37.239335+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(lua_pcall+0x58)[0x55e587dcf468] May 23 21:22:37.239686+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(evalGenericCommand+0x1ff)[0x55e587d743ff] May 23 21:22:37.240007+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(call+0xb1)[0x55e587d09bb1] May 23 21:22:37.240352+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(processCommand+0x5b3)[0x55e587d0aab3] May 23 21:22:37.240687+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(processInputBuffer+0xf8)[0x55e587d1f738] May 23 21:22:37.241010+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(+0x10513c)[0x55e587dbf13c] May 23 21:22:37.241348+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(aeProcessEvents+0x2c2)[0x55e587d02032] May 23 21:22:37.241695+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(aeMain+0x1d)[0x55e587d022bd] May 23 21:22:37.242002+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(main+0x337)[0x55e587cfe307] May 23 21:22:37.242311+00:00 2024 leaf5 INFO database#supervisord: redis6 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f5a6dd21d0a] May 23 21:22:37.242637+00:00 2024 leaf5 INFO database#supervisord: redis6 /usr/bin/redis-server 127.0.0.1:63796(_start+0x2a)[0x55e587cfe83a] May 23 21:22:37.242906+00:00 2024 leaf5 INFO database#supervisord: redis6

Describe the results you expected:

No redis-server crash is expected.

Output of show version:

SONiC Software Version: SONiC-OS-Enterprise_Advanced
Product: Enterprise SONiC Distribution
Distribution: Debian 11.9
Kernel: 5.10.0-21-amd64
Config DB Version: version_4_3_1
Build commit: 5f6ef4fe90
Build date: Fri May 17 13:41:19 UTC 2024

Output of show techsupport:


root@sonic:/# gdb /usr/bin/redis-server core.redis-server.0.ba1a0b8780ea4774b75a47d5ce70532c.1253.1716499357000000
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/redis-server...
(No debugging symbols found in /usr/bin/redis-server)
[New LWP 75]
[New LWP 122]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/redis-server 127.0.0.1:63796                                         '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f5a6dd37087 in kill () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f5a6da82000 (LWP 75))]
(gdb) bt
#0  0x00007f5a6dd37087 in kill () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x000055e587d5ffe5 in bugReportEnd ()
#2  <signal handler called>
#3  0x000055e587d110f9 in sdsMakeRoomFor ()
#4  0x000055e587d11794 in sdscatlen ()
#5  0x000055e587d89fe6 in notifyKeyspaceEvent ()
#6  0x000055e587d51819 in hsetCommand ()
#7  0x000055e587d09bb1 in call ()
#8  0x000055e587d76df9 in luaRedisGenericCommand ()
#9  0x000055e587dd1858 in ?? ()
#10 0x000055e587ddc020 in ?? ()
#11 0x000055e587dd1ef5 in ?? ()
#12 0x000055e587dd11eb in ?? ()
#13 0x000055e587dd20a0 in ?? ()
#14 0x000055e587dcf468 in lua_pcall ()
#15 0x000055e587d743ff in evalGenericCommand ()
#16 0x000055e587d09bb1 in call ()
#17 0x000055e587d0aab3 in processCommand ()
#18 0x000055e587d1f738 in processInputBuffer ()
#19 0x000055e587dbf13c in ?? ()
#20 0x000055e587d02032 in aeProcessEvents ()
#21 0x000055e587d022bd in aeMain ()
#22 0x000055e587cfe307 in main ()
(gdb)

Additional information you deem important (e.g. issue happens only occasionally):

In docker database of SONiC, redis-server debian packages is downloaded from packages.redis.io/deb and installed. I don't find the redis-server-dbgsym file in the internet to debug the line that crashed in sds.c

prabhataravind commented 3 months ago

@sg893052 Could you please provide the SONiC version you are running the tests on and please provide more details on what tests you are running for longevity? Please provide techsupport dump when the issue is seen.