Core dump after owntone command is done scanning the libray (at startup)

hacketiwack commented 5 months ago

With the latest commit of OwnTone Server, I get a core dump each time OwnTone starts at the end of the library scan.

Process 8688 (owntone) of user 0 dumped core.
Stack trace of thread 367:
#0  0x00007e8908aef1ab n/a (/lib/ld-musl-x86_64.so.1 + 0x591ab)
#1  0x0000000000000001 n/a (n/a + 0x0)
ELF object binary architecture: AMD x86-64

I don't know where to start investigating, and any help would therefore be appreciated.

For the record, I tried this previous version which is working fine.

Moreover, there are a few warnings at compile time, which are probably not related:

OwnTone is in my case running on Alpine Linux (in a container).

The versions of installed packages

```bash alpine-baselayout-3.4.3-r2 x86_64 {alpine-baselayout} (GPL-2.0-only) [installed] alpine-baselayout-data-3.4.3-r2 x86_64 {alpine-baselayout} (GPL-2.0-only) [installed] alpine-keys-2.4-r1 x86_64 {alpine-keys} (MIT) [installed] alsa-lib-1.2.10-r0 x86_64 {alsa-lib} (LGPL-2.1-or-later) [installed] aom-libs-3.7.1-r0 x86_64 {aom} (BSD-2-Clause AND custom) [installed] apk-tools-2.14.4-r0 x86_64 {apk-tools} (GPL-2.0-only) [installed] avahi-0.8-r16 x86_64 {avahi} (LGPL-2.1-or-later) [installed] avahi-libs-0.8-r16 x86_64 {avahi} (LGPL-2.1-or-later) [installed] avahi-openrc-0.8-r16 x86_64 {avahi} (LGPL-2.1-or-later) [installed] brotli-libs-1.1.0-r1 x86_64 {brotli} (MIT) [installed] busybox-1.36.1-r19 x86_64 {busybox} (GPL-2.0-only) [installed] busybox-binsh-1.36.1-r19 x86_64 {busybox} (GPL-2.0-only) [installed] busybox-openrc-1.36.1-r19 x86_64 {busybox} (GPL-2.0-only) [installed] c-ares-1.27.0-r0 x86_64 {c-ares} (MIT) [installed] ca-certificates-20240226-r0 x86_64 {ca-certificates} (MPL-2.0 AND MIT) [installed] ca-certificates-bundle-20240226-r0 x86_64 {ca-certificates} (MPL-2.0 AND MIT) [installed] cjson-1.7.17-r0 x86_64 {cjson} (MIT) [installed] confuse-3.3-r4 x86_64 {confuse} (ISC) [installed] curl-8.5.0-r0 x86_64 {curl} (curl) [installed] dbus-1.14.10-r0 x86_64 {dbus} (AFL-2.1 OR GPL-2.0-or-later) [installed] dbus-daemon-launch-helper-1.14.10-r0 x86_64 {dbus} (AFL-2.1 OR GPL-2.0-or-later) [installed] dbus-libs-1.14.10-r0 x86_64 {dbus} (AFL-2.1 OR GPL-2.0-or-later) [installed] dbus-openrc-1.14.10-r0 x86_64 {dbus} (AFL-2.1 OR GPL-2.0-or-later) [installed] ffmpeg-6.1.1-r0 x86_64 {ffmpeg} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] ffmpeg-libavcodec-6.1.1-r0 x86_64 {ffmpeg} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] ffmpeg-libavdevice-6.1.1-r0 x86_64 {ffmpeg} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] ffmpeg-libavfilter-6.1.1-r0 x86_64 {ffmpeg} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] ffmpeg-libavformat-6.1.1-r0 x86_64 {ffmpeg} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] ffmpeg-libavutil-6.1.1-r0 x86_64 {ffmpeg} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] ffmpeg-libpostproc-6.1.1-r0 x86_64 {ffmpeg} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] ffmpeg-libswresample-6.1.1-r0 x86_64 {ffmpeg} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] ffmpeg-libswscale-6.1.1-r0 x86_64 {ffmpeg} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] fontconfig-2.14.2-r4 x86_64 {fontconfig} (MIT) [installed] freetype-2.13.2-r0 x86_64 {freetype} (FTL OR GPL-2.0-or-later) [installed] fribidi-1.0.13-r0 x86_64 {fribidi} (LGPL-2.1-or-later) [installed] glib-2.78.6-r0 x86_64 {glib} (LGPL-2.1-or-later) [installed] glslang-libs-1.3.261.1-r0 x86_64 {glslang} (BSD-3-Clause) [installed] gmp-6.3.0-r0 x86_64 {gmp} (LGPL-3.0-or-later OR GPL-2.0-or-later) [installed] gnutls-3.8.4-r0 x86_64 {gnutls} (LGPL-2.1-or-later) [installed] graphite2-1.3.14-r6 x86_64 {graphite2} (LGPL-2.1-or-later OR MPL-1.1) [installed] harfbuzz-8.3.0-r0 x86_64 {harfbuzz} (MIT) [installed] hwdata-pci-0.377.2-r0 x86_64 {hwdata} (GPL-2.0-or-later OR XFree86-1.1) [installed] ifupdown-ng-0.12.1-r4 x86_64 {ifupdown-ng} (ISC) [installed] json-c-0.17-r0 x86_64 {json-c} (MIT) [installed] lame-libs-3.100-r5 x86_64 {lame} (LGPL-2.0-or-later) [installed] lcms2-2.15-r4 x86_64 {lcms2} (MIT) [installed] libass-0.17.1-r1 x86_64 {libass} (ISC) [installed] libasyncns-0.8-r2 x86_64 {libasyncns} (LGPL-2.0-or-later) [installed] libblkid-2.39.3-r0 x86_64 {util-linux} (LGPL-2.1-or-later) [installed] libbluray-1.3.4-r1 x86_64 {libbluray} (LGPL-2.1-or-later) [installed] libbsd-0.11.7-r3 x86_64 {libbsd} (BSD-3-Clause) [installed] libbz2-1.0.8-r6 x86_64 {bzip2} (bzip2-1.0.6) [installed] libc-utils-0.7.2-r5 x86_64 {libc-dev} (BSD-2-Clause AND BSD-3-Clause) [installed] libcap2-2.69-r1 x86_64 {libcap} (BSD-3-Clause OR GPL-2.0-only) [installed] libcrypto3-3.1.5-r0 x86_64 {openssl} (Apache-2.0) [installed] libcurl-8.5.0-r0 x86_64 {curl} (curl) [installed] libdaemon-0.14-r4 x86_64 {libdaemon} (LGPL-2.1-or-later) [installed] libdav1d-1.3.0-r1 x86_64 {dav1d} (BSD-2-Clause) [installed] libdovi-3.2.0-r1 x86_64 {libdovi} (MIT) [installed] libdrm-2.4.118-r0 x86_64 {libdrm} (MIT) [installed] libevent-2.1.12-r7 x86_64 {libevent} (BSD-3-Clause) [installed] libexpat-2.6.2-r0 x86_64 {expat} (MIT) [installed] libffi-3.4.4-r3 x86_64 {libffi} (MIT) [installed] libflac-1.4.3-r1 x86_64 {flac} (BSD-3-Clause AND GPL-2.0-or-later) [installed] libgcc-13.2.1_git20231014-r0 x86_64 {gcc} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] libgcrypt-1.10.3-r0 x86_64 {libgcrypt} (LGPL-2.1-or-later AND GPL-2.0-or-later) [installed] libgomp-13.2.1_git20231014-r0 x86_64 {gcc} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] libgpg-error-1.47-r2 x86_64 {libgpg-error} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] libhwy-1.0.7-r0 x86_64 {highway} (Apache-2.0) [installed] libidn2-2.3.4-r4 x86_64 {libidn2} (GPL-2.0-or-later OR LGPL-3.0-or-later) [installed] libintl-0.22.3-r0 x86_64 {gettext} (LGPL-2.1-or-later) [installed] libjpeg-turbo-3.0.1-r0 x86_64 {libjpeg-turbo} (BSD-3-Clause AND IJG AND Zlib) [installed] libjxl-0.8.2-r0 x86_64 {libjxl} (Apache-2.0) [installed] libltdl-2.4.7-r3 x86_64 {libtool} (LGPL-2.0-or-later AND GPL-2.0-or-later) [installed] libmd-1.1.0-r0 x86_64 {libmd} (BSD-3-Clause AND BSD-2-Clause AND ISC AND Beerware AND Public Domain) [installed] libmount-2.39.3-r0 x86_64 {util-linux} (LGPL-2.1-or-later) [installed] libncursesw-6.4_p20231125-r0 x86_64 {ncurses} (X11) [installed] libogg-1.3.5-r5 x86_64 {libogg} (BSD-3-Clause) [installed] libopenmpt-0.7.3-r1 x86_64 {libopenmpt} (BSD-3-Clause) [installed] libpciaccess-0.17-r2 x86_64 {libpciaccess} (X11) [installed] libplacebo-6.338.1-r0 x86_64 {libplacebo} (LGPL-2.1-or-later) [installed] libplist-2.3.0-r1 x86_64 {libplist} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] libpng-1.6.40-r0 x86_64 {libpng} (Libpng) [installed] libpulse-16.1-r11 x86_64 {pulseaudio} (LGPL-2.1-or-later) [installed] librist-0.2.10-r0 x86_64 {librist} (BSD-2-Clause) [installed] libsharpyuv-1.3.2-r0 x86_64 {libwebp} (BSD-3-Clause) [installed] libsndfile-1.2.2-r0 x86_64 {libsndfile} (LGPL-2.1-or-later) [installed] libsodium-1.0.19-r0 x86_64 {libsodium} (ISC) [installed] libsrt-1.5.3-r0 x86_64 {libsrt} (MPL-2.0) [installed] libssh-0.10.6-r0 x86_64 {libssh} (LGPL-2.1-or-later BSD-2-Clause) [installed] libssl3-3.1.5-r0 x86_64 {openssl} (Apache-2.0) [installed] libstdc++-13.2.1_git20231014-r0 x86_64 {gcc} (GPL-2.0-or-later AND LGPL-2.1-or-later) [installed] libSvtAv1Enc-1.7.0-r0 x86_64 {svt-av1} (BSD-3-Clause-Clear) [installed] libtasn1-4.19.0-r2 x86_64 {libtasn1} (LGPL-2.1-or-later) [installed] libtheora-1.1.1-r18 x86_64 {libtheora} (BSD-3-Clause) [installed] libunibreak-5.1-r3 x86_64 {libunibreak} (Zlib) [installed] libunistring-1.1-r2 x86_64 {libunistring} (GPL-2.0-or-later OR LGPL-3.0-or-later) [installed] libuuid-2.39.3-r0 x86_64 {util-linux} (BSD-3-Clause) [installed] libva-2.20.0-r0 x86_64 {libva} (MIT) [installed] libvdpau-1.5-r2 x86_64 {libvdpau} (MIT) [installed] libvorbis-1.3.7-r2 x86_64 {libvorbis} (BSD-3-Clause) [installed] libvpx-1.13.1-r0 x86_64 {libvpx} (BSD-3-Clause) [installed] libwebp-1.3.2-r0 x86_64 {libwebp} (BSD-3-Clause) [installed] libwebpmux-1.3.2-r0 x86_64 {libwebp} (BSD-3-Clause) [installed] libwebsockets-4.3.3-r0 x86_64 {libwebsockets} (MIT) [installed] libx11-1.8.7-r0 x86_64 {libx11} (X11) [installed] libxau-1.0.11-r3 x86_64 {libxau} (MIT) [installed] libxcb-1.16-r0 x86_64 {libxcb} (MIT) [installed] libxdmcp-1.1.4-r3 x86_64 {libxdmcp} (MIT) [installed] libxext-1.3.5-r3 x86_64 {libxext} (MIT) [installed] libxfixes-6.0.1-r3 x86_64 {libxfixes} (MIT) [installed] libxml2-2.11.8-r0 x86_64 {libxml2} (MIT) [installed] libzmq-4.3.5-r2 x86_64 {zeromq} (MPL-2.0) [installed] lilv-libs-0.24.22-r0 x86_64 {lilv} (ISC) [installed] linux-pam-1.5.3-r7 x86_64 {linux-pam} (BSD-3-Clause) [installed] mbedtls-2.28.8-r0 x86_64 {mbedtls} (Apache-2.0 OR GPL-2.0-or-later) [installed] mpg123-libs-1.32.3-r0 x86_64 {mpg123} (LGPL-2.1-only) [installed] musl-1.2.4_git20230717-r4 x86_64 {musl} (MIT) [installed] musl-utils-1.2.4_git20230717-r4 x86_64 {musl} (MIT AND BSD-2-Clause AND GPL-2.0-or-later) [installed] ncurses-terminfo-base-6.4_p20231125-r0 x86_64 {ncurses} (X11) [installed] nettle-3.9.1-r0 x86_64 {nettle} (GPL-2.0-or-later OR LGPL-3.0-or-later) [installed] nghttp2-libs-1.58.0-r0 x86_64 {nghttp2} (MIT) [installed] numactl-2.0.16-r4 x86_64 {numactl} (LGPL-2.1-only) [installed] onevpl-libs-2023.3.1-r2 x86_64 {onevpl} (MIT) [installed] openrc-0.52.1-r2 x86_64 {openrc} (BSD-2-Clause) [installed] opus-1.4-r0 x86_64 {opus} (BSD-3-Clause) [installed] orc-0.4.34-r0 x86_64 {orc} (BSD-2-Clause) [installed] p11-kit-0.25.3-r0 x86_64 {p11-kit} (BSD-3-Clause) [installed] pcre2-10.42-r2 x86_64 {pcre2} (BSD-3-Clause) [installed] protobuf-c-1.4.1-r7 x86_64 {protobuf-c} (BSD-2-Clause) [installed] rav1e-libs-0.6.6-r2 x86_64 {rav1e} (BSD-2-Clause custom) [installed] readline-8.2.1-r2 x86_64 {readline} (GPL-2.0-or-later) [installed] scanelf-1.3.7-r2 x86_64 {pax-utils} (GPL-2.0-only) [installed] sdl2-2.28.5-r0 x86_64 {sdl2} (Zlib) [installed] serd-libs-0.32.0-r0 x86_64 {serd} (ISC) [installed] shaderc-2023.7-r0 x86_64 {shaderc} (Apache-2.0) [installed] shadow-4.14.2-r0 x86_64 {shadow} (BSD-3-Clause) [installed] skalibs-2.14.0.1-r0 x86_64 {skalibs} (ISC) [installed] sord-libs-0.16.16-r0 x86_64 {sord} (ISC) [installed] soxr-0.1.3-r7 x86_64 {soxr} (LGPL-2.1-or-later) [installed] speexdsp-1.2.1-r2 x86_64 {speexdsp} (BSD-3-Clause) [installed] spirv-tools-1.3.261.1-r0 x86_64 {spirv-tools} (Apache-2.0) [installed] sqlite-3.44.2-r0 x86_64 {sqlite} (blessing) [installed] sqlite-libs-3.44.2-r0 x86_64 {sqlite} (blessing) [installed] sratom-0.6.16-r0 x86_64 {sratom} (ISC) [installed] ssl_client-1.36.1-r19 x86_64 {busybox} (GPL-2.0-only) [installed] tdb-libs-1.4.9-r0 x86_64 {tdb} (LGPL-3.0-or-later) [installed] udev-init-scripts-openrc-35-r1 x86_64 {udev-init-scripts} (GPL-2.0-only) [installed] utmps-libs-0.1.2.2-r0 x86_64 {utmps} (ISC) [installed] v4l-utils-libs-1.24.1-r1 x86_64 {v4l-utils} (LGPL-2.0-or-later) [installed] vidstab-1.1.1-r0 x86_64 {vidstab} (GPL-2.0-or-later) [installed] vulkan-loader-1.3.261.1-r0 x86_64 {vulkan-loader} (Apache-2.0) [installed] wayland-libs-client-1.22.0-r4 x86_64 {wayland} (MIT) [installed] x264-libs-0.164_git20231001-r0 x86_64 {x264} (GPL-2.0-or-later) [installed] x265-libs-3.5-r4 x86_64 {x265} (GPL-2.0-or-later) [installed] xvidcore-1.3.7-r2 x86_64 {xvidcore} (GPL-2.0-or-later) [installed] xz-libs-5.4.5-r0 x86_64 {xz} (GPL-2.0-or-later AND Public-Domain AND LGPL-2.1-or-later) [installed] zimg-3.0.5-r2 x86_64 {zimg} (WTFPL) [installed] zix-libs-0.4.2-r0 x86_64 {zix} (ISC) [installed] zlib-1.3.1-r0 x86_64 {zlib} (Zlib) [installed] ```

ejurgensen commented 5 months ago

Can you run it in gdb or Valgrind? E.g. "sudo gdb --args /path/to/owntone -f"

hacketiwack commented 5 months ago

I will do that. Thanks for the tip.

hacketiwack commented 5 months ago

Strangely enough, running OwnTone with gdb does not make it crash. I didn't compile OwnTone with debug symbols for now.

However, using valgrind I get systematically a crash. I attached the Valgrind logs.

valgrind.log

ejurgensen commented 5 months ago

Oh boy, so much crap in that Valgrind log. However, the real error was in fact hidden in it:

==341== Thread 17:
==341== Invalid read of size 1
==341==    at 0x48B4D5A: strcmp (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==341==    by 0x17AA81: speaker_get_byaddress (player.c:2620)
==341==    by 0x1B2570: command_cb_sync (commands.c:85)
==341==    by 0x1B2570: command_cb (commands.c:142)
==341==    by 0x77175FC: ??? (in /usr/lib/libevent-2.1.so.7.0.1)
==341==    by 0x7717D25: event_base_loop (in /usr/lib/libevent-2.1.so.7.0.1)
==341==    by 0x17B21A: player (player.c:3808)
==341==    by 0x405C22D: ??? (in /lib/ld-musl-x86_64.so.1)
==341==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

As far as I can tell, this means that player_speaker_get_byaddress() is called with a NULL argument, but from the code I can't see how that can happen. I think I will add some extra logging and ask you to retest.

ejurgensen commented 5 months ago

I see that there already should be some relevant logging. If you run normally, but with debug level, you should see this before the crash:

DPRINTF(E_DBG, L_HTTPD, "Checking if client '%s' is a speaker\n", hreq->peer_address);

Is that the case? Does it say that the address is NULL?

hacketiwack commented 5 months ago

Yes, I confirm that this message appears in the logs.

[2024-06-22 06:36:18] [DEBUG]    httpd: Checking if client '(null)' is a speaker

ejurgensen commented 5 months ago

I think it should be fixed now, thanks for helping find the culprit

hacketiwack commented 5 months ago

Amazing. It works now. Thanks for the quick fix :-)

owntone / owntone-server

Core dump after owntone command is done scanning the libray (at startup) #1771