Add new music backend

[lmichaelis]

This PR adds a new, experimental music backend recently created at GothicKit/dmusic. It can be enabled by changing the "Soundprovider" option in the sound settings to "GothicKit (Experimental)".

Note @Try: I am having really weird multithreading issues on Fedora. It's likely they are not caused by OpenGothic (since std::recursive_mutex should be fine) but because of them I am having a hard time properly testing everything. It should be mostly okay now though. (specifically: the Kernel keeps semi-predictably sending SIGKILL to the process making the cause impossible to easily debug)

[Try]

the Kernel keeps semi-predictably sending SIGKILL to the process

What do you mean by "semi-predictably"? Do you have a steps to reproduce?

[lmichaelis]

the Kernel keeps semi-predictably sending SIGKILL to the process

What do you mean by "semi-predictably"? Do you have a steps to reproduce?

Nope, I don't. It just starts happening randomly in the codebase and then persists at the same that until I debug it enough, then it starts working again. For example, the game plays the gamestart sound bite which requires it to load a WAV from the resources. Doing that, you lock a Resources-internal mutex, which I narrowed the crash down to. As soon as a tried to figure out where in libc(++) it fails by stepping into it with a debugger it stops crashing any everything is fine again. When I restart, the crash is then gone.

This happened in different places too but when and when it happens is a mystery to me.

[Nindaleth]

Hi @lmichaelis, I'm having issues trying to compile your branch, could I do something better?

Checked out lmichaelis/feat/new-music-system + git pull --recurse-submodules and I get these ZenKit compilation errors: https://pastebin.com/g6uARWui

This is on Fedora 40, gcc (GCC) 14.0.1 20240411

[lmichaelis]

Hi @lmichaelis, I'm having issues trying to compile your branch, could I do something better?

Checked out lmichaelis/feat/new-music-system + git pull --recurse-submodules and I get these ZenKit compilation errors: https://pastebin.com/g6uARWui

This is on Fedora 40, gcc (GCC) 14.0.1 20240411

Heyo, looks like your compiler is trying to compile for C++14, rather than >= C++17. Have you tried resetting your CMake cache and recompiling?

[Nindaleth]

I've disabled ccache, removed the OpenGothic build directory entirely, I even tried to recreate with cmake -B build -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_CXX_STANDARD=17 -DCXX_STANDARD=17, but the issue stays the same.

Are these missing fuctions specifically C++17 additions? From my log it may be misleading but I think there's no mention of the C++ standard used, it's just by chance that the current compiler major version is 14, this GCC implements C++20 except for modules and almost all of C++23, should be new enough:

$ ll /usr/include/c++/
drwxr-xr-x. root root   4 B  Fri May  3 10:47:44 2024 ./
drwxr-xr-x. root root 6.9 KB Fri May  3 10:47:33 2024 ../
drwxr-xr-x. root root 1.9 KB Fri May  3 10:38:48 2024 14/

$ rpm -qf /usr/include/c++/14
libstdc++-devel-14.0.1-0.15.fc40.x86_64

$ whatdepends libstdc++ | grep ^gcc
gcc-c++-0:14.0.1-0.15.fc40.x86_64
gcc-gdb-plugin-0:14.0.1-0.15.fc40.x86_64
gcc-plugin-annobin-0:14.0.1-0.15.fc40.x86_64

[Nindaleth]

Maybe it's this? https://gcc.gnu.org/gcc-14/porting_to.html:

Header dependency changes

Some C++ Standard Library headers have been changed to no longer include other headers that were being used internally by the library. As such, C++ programs that used standard library components without including the right headers will no longer compile.

The following headers are used less widely in libstdc++ and may need to be included explicitly when compiling with GCC 14:

\<algorithm> (for std::copy_n, std::find_if, std::lower_bound, std::remove, std::reverse, std::sort etc.) \<cstdint> (for std::int8_t, std::int32_t etc.)

[Nindaleth]

Indeed, sticking #include <algorithm> into Internal.hh and into Mesh.hh (as a quick hackfix) does allow the compilation to finish.

[Nindaleth]

Sorry for posting too much, one last post. Going to Valley of Mines and switching the sound provider results in this:

Switching music provider to 'GothicKit'
[dmusic] DmBand: Downloading instruments for band 'Ingame'
[dmusic] DmRiff: Chunk LIST:[adtl] not fully parsed, 16 bytes remaining
[dmusic] DmRiff: Chunk LIST:[adtl] not fully parsed, 16 bytes remaining
[dmusic] DmBand: Instrument patch 1:10 not found in band 'DLS_Metronom'
[dmusic] DmBand: Instrument patch 1:19 not found in band 'DLS_Rare'
[dmusic] DmBand: Downloading instruments for band 'Ingame'
[dmusic] DmBand: Instrument patch 1:10 not found in band 'DLS_Metronom'
[dmusic] DmBand: Instrument patch 1:19 not found in band 'DLS_Rare'
[dmusic] DmLoader: Automatic download of segment 'owd_daystd.sgt' succeeded
malloc(): unaligned tcache chunk detected

Thread 5 "threaded-ml" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe56006c0 (LWP 69657)]
(anonymous namespace)::LoadSamples<(FmtType)1> (dstSamples=..., dstOffset=<optimized out>, src=<optimized out>, srcOffset=<optimized out>, srcChans=<optimized out>, srcStep=2, samples=965) at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/core/voice.cpp:236
236             al::LoadSampleArray<Type>(dst+dstOffset, s, srcStep, samples);
Missing separate debuginfos, use: dnf debuginfo-install dbus-libs-1.14.10-3.fc40.x86_64 elfutils-libelf-0.191-4.fc40.x86_64 expat-2.6.2-1.fc40.x86_64 flac-libs-1.4.3-4.fc40.x86_64 fmt-10.2.1-4.fc40.x86_64 glibc-2.39-8.fc40.x86_64 gsm-1.0.22-6.fc40.x86_64 lame-libs-3.100-17.fc40.x86_64 libX11-1.8.9-1.fc40.x86_64 libX11-xcb-1.8.9-1.fc40.x86_64 libXau-1.0.11-6.fc40.x86_64 libXcursor-1.2.1-7.fc40.x86_64 libXfixes-6.0.1-3.fc40.x86_64 libXrender-0.9.11-6.fc40.x86_64 libasyncns-0.8-28.fc40.x86_64 libatomic-14.0.1-0.15.fc40.x86_64 libcap-2.69-8.fc40.x86_64 libdrm-2.4.120-3.fc40.x86_64 libedit-3.1-50.20230828cvs.fc40.x86_64 libffi-3.4.4-7.fc40.x86_64 libgcc-14.0.1-0.15.fc40.x86_64 libogg-1.3.5-8.fc40.x86_64 libsndfile-1.2.2-2.fc40.x86_64 libstdc++-14.0.1-0.15.fc40.x86_64 libvorbis-1.3.7-10.fc40.x86_64 libwayland-client-1.22.0-3.fc40.x86_64 libxcb-1.17.0-1.fc40.x86_64 libxshmfence-1.3.2-3.fc40.x86_64 libzstd-1.5.6-1.fc40.x86_64 llvm-libs-18.1.1-1.fc40.x86_64 mangohud-0.7.1-3.fc40.x86_64 mesa-vulkan-drivers-24.0.6-2.fc40.x86_64 mpg123-libs-1.31.3-4.fc40.x86_64 ncurses-libs-6.4-12.20240127.fc40.x86_64 opus-1.5.1-1.fc40.x86_64 systemd-libs-255.4-1.fc40.x86_64 vulkan-loader-1.3.280.0-1.fc40.x86_64 xz-libs-5.4.6-3.fc40.x86_64 zlib-ng-compat-2.1.6-2.fc40.x86_64
(gdb) bt
#0  (anonymous namespace)::LoadSamples<(FmtType)1> (dstSamples=..., dstOffset=<optimized out>, src=<optimized out>, srcOffset=<optimized out>, srcChans=<optimized out>, srcStep=2, samples=965) at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/core/voice.cpp:236
#1  (anonymous namespace)::LoadSamples (dstSamples=..., dstOffset=<optimized out>, src=<optimized out>, srcOffset=<optimized out>, srcType=<optimized out>, srcChans=FmtStereo, srcStep=2, samples=965) at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/core/voice.cpp:254
#2  0x00007ffff7bbb30c in (anonymous namespace)::LoadBufferCallback (buffer=<optimized out>, numCallbackSamples=<optimized out>, sampleType=<optimized out>, sampleChannels=<optimized out>, srcStep=<optimized out>, samplesLoaded=<optimized out>, samplesToLoad=965, voiceSamples=...)
    at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/core/voice.cpp:318
#3  Voice::mix (this=0x20dd8f8, vstate=Voice::Stopping, Context=Context@entry=0x1aadce0, deviceTime=..., deviceTime@entry=std::chrono::duration = { 60433333333ns }, SamplesToDo=SamplesToDo@entry=1024) at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/core/voice.cpp:685
#4  0x00007ffff7b8149a in (anonymous namespace)::ProcessContexts (device=0x20dd2e8, SamplesToDo=<optimized out>) at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/alc/alu.cpp:1868
#5  DeviceBase::renderSamples (this=this@entry=0x111abe0, numSamples=<optimized out>) at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/alc/alu.cpp:2073
#6  0x00007ffff7b828d8 in DeviceBase::renderSamples (this=<optimized out>, outBuffer=0x7fffdc000040, numSamples=1024, frameStep=<optimized out>) at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/alc/alu.cpp:2132
#7  0x00007ffff7b9b3eb in (anonymous namespace)::PulsePlayback::streamWriteCallback (this=0x1167610, stream=0x1282810, nbytes=0) at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/alc/backends/pulseaudio.cpp:680
#8  operator() (__closure=0x0, stream=0x1282810, nbytes=<optimized out>, pdata=0x1167610) at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/alc/backends/pulseaudio.cpp:979
#9  _FUN () at /home/black_fox/src/OpenGothic/lib/Tempest/Engine/thirdparty/openal-soft/alc/backends/pulseaudio.cpp:979
#10 0x00007ffff4882c25 in pa_command_request () from /lib64/libpulse.so.0

[lmichaelis]

Indeed, sticking #include <algorithm> into Internal.hh and into Mesh.hh (as a quick hackfix) does allow the compilation to finish.

Thanks! I'll fix it asap.

Sorry for posting too much, one last post. Going to Valley of Mines and switching the sound provider results in this

Could you turn on trace logging for dmusic in main.cpp and post the output?

[Nindaleth]

Could you turn on trace logging for dmusic in main.cpp and post the output?

Sure, here it is:

Switching music provider to 'GothicKit'
[dmusic] DmLoader: Loading segment 'owd_daystd.sgt'
[dmusic] DmLoader: Loading style '_OldWorld.sty'
[dmusic] DmBand: Downloading instruments for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Bass.dls'
[dmusic] DmRiff: Chunk LIST:[adtl] not fully parsed, 16 bytes remaining
[dmusic] DmRiff: Chunk LIST:[adtl] not fully parsed, 16 bytes remaining
[dmusic] DmBand: DLS instrument 'Bass_L' assigned to channel 0 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Bass_R' assigned to channel 1 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Flute.dls'
[dmusic] DmBand: DLS instrument 'Flute_L' assigned to channel 2 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Flute_R' assigned to channel 3 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Harp.dls'
[dmusic] DmBand: DLS instrument 'Harp_Mono' assigned to channel 4 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Strings.dls'
[dmusic] DmBand: DLS instrument 'Strings_L' assigned to channel 5 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Strings_R' assigned to channel 6 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Strings_L' assigned to channel 7 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Strings_R' assigned to channel 8 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Metronom.dls'
[dmusic] DmBand: Instrument patch 1:10 not found in band 'DLS_Metronom'
[dmusic] DmLoader: Loading DLS collection 'DLS_Piano.dls'
[dmusic] DmBand: DLS instrument 'Piano_L' assigned to channel 10 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Piano_R' assigned to channel 11 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Guitar.dls'
[dmusic] DmBand: DLS instrument 'Guitar_Mono' assigned to channel 12 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Brass.dls'
[dmusic] DmBand: DLS instrument 'Brass_L' assigned to channel 13 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Brass_R' assigned to channel 14 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Brass_L' assigned to channel 15 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Brass_R' assigned to channel 16 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Daduk.dls'
[dmusic] DmBand: DLS instrument 'Daduk_L' assigned to channel 17 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Daduk_R' assigned to channel 18 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Alpenhorn.dls'
[dmusic] DmBand: DLS instrument 'Alpenhorn_L' assigned to channel 19 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Alpenhorn_R' assigned to channel 20 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Percussions.dls'
[dmusic] DmBand: DLS instrument 'Percussion_L' assigned to channel 21 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Percussion_R' assigned to channel 22 for band 'Ingame'
[dmusic] DmLoader: Loading DLS collection 'DLS_Rare.dls'
[dmusic] DmBand: DLS instrument 'Rare_L' assigned to channel 23 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Rare_R' assigned to channel 24 for band 'Ingame'
[dmusic] DmBand: Instrument patch 1:19 not found in band 'DLS_Rare'
[dmusic] DmLoader: Loading DLS collection 'DLS_Organ.dls'
[dmusic] DmBand: DLS instrument 'Organ_L' assigned to channel 26 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Organ_R' assigned to channel 27 for band 'Ingame'
[dmusic] DmBand: Downloading instruments for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Bass_L' assigned to channel 0 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Bass_R' assigned to channel 1 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Flute_L' assigned to channel 2 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Flute_R' assigned to channel 3 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Harp_Mono' assigned to channel 4 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Strings_L' assigned to channel 5 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Strings_R' assigned to channel 6 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Strings_L' assigned to channel 7 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Strings_R' assigned to channel 8 for band 'Ingame'
[dmusic] DmBand: Instrument patch 1:10 not found in band 'DLS_Metronom'
[dmusic] DmBand: DLS instrument 'Piano_L' assigned to channel 10 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Piano_R' assigned to channel 11 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Guitar_Mono' assigned to channel 12 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Brass_L' assigned to channel 13 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Brass_R' assigned to channel 14 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Brass_L' assigned to channel 15 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Brass_R' assigned to channel 16 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Daduk_L' assigned to channel 17 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Daduk_R' assigned to channel 18 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Alpenhorn_L' assigned to channel 19 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Alpenhorn_R' assigned to channel 20 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Percussion_L' assigned to channel 21 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Percussion_R' assigned to channel 22 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Rare_L' assigned to channel 23 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Rare_R' assigned to channel 24 for band 'Ingame'
[dmusic] DmBand: Instrument patch 1:19 not found in band 'DLS_Rare'
[dmusic] DmBand: DLS instrument 'Organ_L' assigned to channel 26 for band 'Ingame'
[dmusic] DmBand: DLS instrument 'Organ_R' assigned to channel 27 for band 'Ingame'
[dmusic] DmLoader: Automatic download of segment 'owd_daystd.sgt' succeeded
malloc(): unaligned tcache chunk detected

[lmichaelis]

Sure, here it is:

Thanks @Nindaleth, but I am unable to replicate the issue. Which distribution of Gothic are you using and how exactly do you trigger the crash? A short video might be helpful for illustration.

[Nindaleth]

Which distribution of Gothic are you using

It's the GOG's Gothic 2 Gold in English, installed from setup_gothic_2_gold_2.7_(14553).exe version.

how exactly do you trigger the crash?

I have MARVIN'd my way into the very beginning of Valley of Mines (it's very easy for me to hear the difference between vanilla and OG), saved there. Then I just load the game and try to switch, nothing really special that could be caught on video.

I start the game like this (then run and no debuginfod) ~/src/OpenGothic $ LD_LIBRARY_PATH=./build/opengothic/ gdb --args /home/black_fox/src/OpenGothic/build/opengothic/Gothic2Notr -g '/home/black_fox/Games/gothic-ii-gold-edition/drive_c/GOG Games/Gothic 2 Gold/' -save 19

save_slot_19.zip

I could upload some of the Gothic's data files into a temporary online storage if that'd help?

[lmichaelis]

Thanks for the explanation. I still can't get this crash to happen though. I do have the GOG version but it does not yield that particular crash. I do still have issues with futexes (at least in debug and debug-release modes) which cause the kernel to kill the OpenGothic process.

I could upload some of the Gothic's data files into a temporary online storage if that'd help?

Not really, but you could post the sha256sum of OWD_DayStd.sgt and _OldWorld.sty just to compare. Here are mine:

22920c08480e6b20a41bcf566cb6720fb6d7898510d92b45c4a439c891a0761c  /mnt/games/Gothic/Gothic26DE/_work/Data/Music/newworld/OWD_DayStd.sgt
045177911583ab7e8f397ed40a0b62bd0ae24bbfa3e7d73b2bd6a67c0eb97b07  /mnt/games/Gothic/Gothic26DE/_work/Data/Music/newworld/_OldWorld.sty

[Try]

Bump zenkit and dmusic

FYI: I've bumped ZenKit on master, so you can rebase

[Nindaleth]

Not really, but you could post the sha256sum of OWD_DayStd.sgt and _OldWorld.sty just to compare. Here are mine:

Files on my install are matching your sha256sums completely. My issue was elsewhere and it was self-inflicted.

Since the original issues compiling I had tried setting a newer C++ standard in OpenAL vendored lib in case it would force the rest of the codebase onto that older standard - which turned out to be bogus because it was caused by a change in behaviour of GCC 14. That was the sole source of trouble; after reverting this change and recompiling I don't have get any more crashes on switching the audio provider. There's still a few mentions of missing instrument patches in the log just like before, but with the tracing on I'm now also getting lots of

[dmusic] DmPerformance(Message): time=41009 type=control-change channel=1 control=11 value=0.454053

and the music plays, audibly different compared to the OG sound provider. Hooray!