Music crackling/popping

[Novicek]

Preliminary checks

• [X] I've checked that there aren't other open issues on the same topic.
• [X] I've checked that this issue is reproducible on the latest snapshot build.

Platform

Windows

Describe the bug

Most of the music in this version of the game has a very noticeable crackle, which can be noticed only few seconds into the main menu music for example, but happens during all the music in game I heard so far.

Of course I've tried first if the issue is on my side, but was not able to resolve it: My audio drivers are up to date. I have no audio enhancements on my audio device enabled. My format is set to 48000Hz 24Bits and I've tried almost every possible setting too (reducing channels to 16Bits, changing sample rate to 44100Hz or even 96000Hz).

However, switching from my headphones to my speakers, I didn't hear any crackle, so it is most likely on my side. I am still putting this here in case of a suggestion and also because the same music does not have crackle in the other versions of the game (GOG version for DOS/Windows are both fine), so it still possibly has something to do with the way the playback works in this project.

Save file

Save file not required to reproduce the issue.

Additional info

Playing the track (homm2_41.ogg) itself via a media player in Windows, I do not hear the crackle either, even with volume set to much higher than in game.

[zenseii]

Hey, @Novicek. So if I'm not mistaken your files are in .ogg format. I haven't heard about this issue with .ogg files before. You could try to download the higher quality .flac pack from the Heroes 2 download page on GOG among the extras and use that with fheroes2.

There's a script that comes with fheroes2 that automatically renames all the music files for you and imports them from the .zip archive.

[Novicek]

Hey @zenseii, thanks for the reply. I tried what you suggested and the result is still the same, I get crackle when the music plays through fheroes2.

Just to confirm I did it right, I downloaded the zip with flac files, ran the resource_extraction_toolset.bat, dropped the zip in there, which created a music.flac folder. I then backed the original music folder and renamed music.flac to music.

[zenseii]

Just to confirm I did it right

Yep, that's the right procedure.

Are you testing on the latest snapshot or version 1.0? Not that it should matter afaik.

What are the specs of your PC? I've had intermittent crackling because my PC was really old, but only when moving a hero on the adventure map, while you said this happens to you regularly on any screen?

[Novicek]

I tested on the snapshot before I posted, but then reverted to 1.0.0. I also experienced with older builds and it's the same everywhere.

I built my PC January 2021, with a Ryzen9 5900x, RTX 3070, 32GB RAM on the Gigabyte B550 AORUS ELITE motherboard. I don't get performance issues, it's just the music. Most prominent are the town and main menu music, I hear it a lot less on the adventure map and combat screen. Maybe I could record a video to demonstrate.

[zenseii]

In other words the PC specs is not the problem. Have you tried different headsets?

[Novicek]

I just did on my old earbuds. It's still there, though I hear it a bit less.

[Districh-ru]

The issue is in SDL2 (or maybe in the way we use it). When I put the SDL2.dll file from new version (1.0 or latest commit) I get the cracks and also the frequency range of music playback is reduced to 11 kHz (as the game uses 22050 sampling frequency):

And when I put SDL2.dll file from v.0.9.21 all is OK:

[Districh-ru]

In audio.cpp we set: freq = 22050;. I've tried to change it to 44100 and this issue has gone... But checking the original sounds I've found that they have some, maybe mirrored, frequencies. The original sounds sample rate is 22050 Hz, so its frequency range is up to 11025 Hz, and we have frequencies up to 16000 Hz: But these 11025-16000 Hz frequencies are about -20 dB, so it may be ignored for now. Also when I used freq 96000 or 192000 Hz I sometimes had cracks in the game sounds, but the music was fine. So the SDL2 re-sampler might have some issues.

@oleg-derevenetz, @ihhub, could you please check this sound playback issue when you have time. Maybe we have to downgrade/upgrade the SDL2 or change the freq to 44100. But this may be not good for all platforms.

[oleg-derevenetz]

Maybe we have to downgrade/upgrade the SDL2 or change the freq to 44100. But this may be not good for all platforms.

We can't upgrade it because we already build the latest available release (2.26.1) for Windows. There were serious issues with their resampler in previous version(s), so I have to downgrade the SDL to 2.0.22 (IIRC), but we can't do this forever. In theory, we can change the frequency to 44100 for Windows only - at least temporarily.

[Districh-ru]

In theory, we can change the frequency to 44100 for Windows only - at least temporarily.

This could be a temporary solution. Taking into account that music in OGG files has 48000 Hz sampling rate, IMHO it is better to have this sampling rate in the game. But with current SDL2.dll this sampling rate causes cracks in the game sounds. :( With 44100 Hz I did not notice any cracks in the sound. This frequency is also good for music files, as the highest frequency in them is about 20.5 kHz. With SDL 2.0.22 the game always gives me the warning: Audio::Init: Audio frequency is initialized as 192000 instead of 22050. So downgrading the SDL will make the game to use the system sample rate. Let's hope that in future versions of SDL the resampler will be fixed.

[oleg-derevenetz]

Support for the libsamplerate was added recently in the sdl2 vcpkg port, and building sdl2 with samplerate feature may resolve this issue, because libsamplerate will be used for resampling instead of the SDL built-in resampler implementation. However vcpkg team (or maybe I will if I succeed) will need to resolve this issue first, because glib is mandatory to build the FluidSynth plugin that is used by the SDL_mixer to play MIDI files.

[ihhub]

I have a feeling we are trying to fix issues in SDL library itself :( If this issue is Windows specific should we (possibly not now) use Windows OS libraries to play midi and music files, like DirectMusic?

[oleg-derevenetz]

@ihhub

use Windows OS libraries to play midi and music files, like DirectMusic?

We already tried to use Windows native MIDI (SDL_mixer uses it if no FluidSynth plugin or soundfont is present) and we all know that there is a ton of problems with it - it is slow and have issues with volume, that's why we moved to the FluidSynth at the time. Also SDL uses its own resampler implementation (and not the Windows resampler, although it can be used in the SDL code) because native resampler is even worse and brings even more distortion :)

I think it's better to spend relatively little time choosing the right plugins/libs for SDL instead of wasting time on some home-grown non-potable implementation that still will need to use the same libs at the end :)

P.S. Ahaha, they turned native Windows resampler on back in November, so now you hear the results of native Windows resampler :) It seems that "just use Windows native libs" is not an option in any case.

[oleg-derevenetz]

Although I was able to build SDL2 with libsamplerate support for Windows, I'm not sure how it will work and if libsamplerate resampler will be involved at all, because it seems that currently SDL2 uses Windows resampler instead of its own (and maybe instead of libsamplerate too) and it is hardcoded. So let's begin from a simpler approach: just set the audio frequency to 44100 for Windows only.

@Novicek @Districh-ru could you please test the build from #6455 with all supported music types (including OGG, MP3, FLAC and MIDI) as well as in-game sounds for these distortions?

[Novicek]

@oleg-derevenetz Tested all sound formats (though MIDI didn't give me crackle even beforehand far as I could tell) and all seems fine.

[Districh-ru]

@oleg-derevenetz, I was busy yesterday, so could not test the build. I've tested current build and found that the sound clicks are almost inaudible, but they are still present in music (OGG, MP3, FLAC) - in spectrogram they are vertical lines: But this is better, than with 22050 Hz sample rate:

And knowing this I'll prefer to use SDL2.dll v. 2.0.22. :(

For long time I've used WinAmp plugin 'YASAPI/NT' (https://sourceforge.net/projects/out-yasapi/), which uses WASAPI. And this plugin with default configuration also had some distortions, including mirrored frequencies. But in plugin configuration there is a setting "SRC Default Quality", enabling which made WinAmp sound perfect - without distortions and mirrored frequencies.

Maybe the authors of SDL should use the same setting to avoid sound distortions of resampler.

[oleg-derevenetz]

Hi @Districh-ru since you already have the infrastructure in place, could you please try the DLLs from this PR:

https://github.com/fheroes2/fheroes2-prebuilt-deps/pull/28

and tell is it makes any difference? Please note that there is one new DLL (samplerate.dll) goes in addition.

[Districh-ru]

Hi @oleg-derevenetz, I tried these DLLs and found that there is no difference with the latest master build. Small clicks still present in music.

[oleg-derevenetz]

@Districh-ru OK, thanks! I also suspected that there will be no difference, because the use of native WASAPI resampler is currently hardcoded in SDL.

[oleg-derevenetz]

@Districh-ru

Maybe the authors of SDL should use the same setting to avoid sound distortions of resampler.

What this setting does is enable the use of the AUDCLNT_STREAMFLAGS_SRC_DEFAULT_QUALITY flag. This flag is already used in SDL:

    /* we've gotten reports that WASAPI's resampler introduces distortions, but in the short term
       it fixes some other WASAPI-specific quirks we haven't quite tracked down.
       Refer to bug #6326 for the immediate concern. */
#if 0
    this->spec.freq = waveformat->nSamplesPerSec;  /* force sampling rate so our resampler kicks in, if necessary. */
#else
    /* favor WASAPI's resampler over our own */
    if (this->spec.freq != waveformat->nSamplesPerSec) {
        streamflags |= (AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM | AUDCLNT_STREAMFLAGS_SRC_DEFAULT_QUALITY);
        waveformat->nSamplesPerSec = this->spec.freq;
        waveformat->nAvgBytesPerSec = waveformat->nSamplesPerSec * waveformat->nChannels * (waveformat->wBitsPerSample / 8);
    }
#endif

And still the results are not good.

[Districh-ru]

So the issue is in some other place. Let's hope that the developers of SDL could fix it (https://github.com/libsdl-org/SDL/issues/6326, https://github.com/libsdl-org/SDL/issues/5538)

And maybe we should leave this issue opened as the bug still exists.

[oleg-derevenetz]

And maybe we should leave this issue opened as the bug still exists.

This bug exists in SDL indeed, but we applied a workaround which works good enough for an ordinary user. However, so be it.

[ihhub]

Hi @oleg-derevenetz and @Districh-ru , as pointed out by a person in Discord it might be that our soundfont is corrupted. This is a reply we got:

I redownloaded GeneralUser GS 1.471 from http://www.schristiancollins.com/generaluser.php
and it does not contain the same clicks as fheroes2.sf3
the soundfont itself seem to be corrupted 

[oleg-derevenetz]

Hi @ihhub but the soundfont is not used to play external (mp3, ogg, whatever) music files at all. It is used only for midi.

[ihhub]

Hi @oleg-derevenetz , the person refers to midi. I will double check with them.

[oleg-derevenetz]

@ihhub SF3 format contains samples in some packed format (like MP3) while SF2 format comtains WAV samples (that's why it is much bigger). Maybe it is some resampling-related issue. Anyway, I don't hear any artifacts in MIDI mode as well.

[Districh-ru]

Hi @ihhub, I tested MIDI play in Windows and Android versions and did not notice any cracks in sound.

[autoexecbatman]

But somebody did post an example with clicks present on discord . https://discord.com/channels/733093692860137523/906187653144268800/1084068036882993212 Can you hear the clicks in this video ?

Also you can just open fheroes2.sf3 using Polyphone soundfont editor and check for yourself. Some of the samples compressed in this soundfont contain clicks. If it is used to play the midi in game then some notes on some instruments will contain those clicks maybe not audible to every person or some speakers/headphones.

[oleg-derevenetz]

Well, I don't hear clicks in SF3 samples. But if someone do, then they are welcome to offer a PR with another soundfont file with appropriate set of instruments (and not very big).

[autoexecbatman]

@oleg-derevenetz https://discord.com/channels/733093692860137523/906187653144268800/1084068036882993212 just curious, can you hear the clicks in this video another user provided ?

[oleg-derevenetz]

@autoexecbatman I don't have a Discord account and do not participate in fheroes2 Discord channel. I just don't like all these messengers because they mostly distract me from thinking and work :)

[autoexecbatman]

@oleg-derevenetz Oh ok . Anyway long story short. http://www.schristiancollins.com/generaluser.php [GeneralUser GS 1.471] - soundfont does not contain these clicks . I think it is the same soundfont that fheroes.sf3 was made of .

[oleg-derevenetz]

@autoexecbatman Earlier in this issue I mentioned that SF3 format is different from SF2, because samples are stored in compressed form in SF3. Original SF2 file is too big even if all unused instruments are removed from it.

[autoexecbatman]

@oleg-derevenetz ok I get it. So I can try make it .sf3 the same size by removing the instruments like you did already. That maybe somehow introduced those clicks.

[Districh-ru]

@ihhub, @autoexecbatman, @oleg-derevenetz, After some tests in the headphones I noticed the cracks and clicks when the music volume is set to 10. Previously I checked it at 7 volume and it is all OK. IMHO - it is a bug of simple resampler (?or decompressor?), which tries to interpolate the sound wave and sometimes the interpolation gives the value more than the maximum sample amplitude and if the Volume is already set to maximum then such value is clipped. And some "super smart" sound cards do not only clip the sound, but also "to prevent speakers from damage" sets the value of the wave to 0 and we get the pulse from the maximum amplitude to zero. I've encountered such "expensive" DAC at my work. So to prevent possible sound clipping and to reduce the THD I never set the volume more than 90-95% of maximum.

The Polyphone soundfont editor also gives me clicks at the maximum amplitude of samples. But OpenMPT, which can also play the samples from soundfonts, does not produce clicks...

[autoexecbatman]

@Districh-ru The clicks are burned inside the samples .

[autoexecbatman]

I converted again like I said . from *.sf2 to *.sf3 removing all the instruments to match fheroes2.sf2 And now it produces clicks !

[autoexecbatman]

The problem is converting from *.sf2 to *.sf3 without introducing clicks.

[Districh-ru]

@Districh-ru The clicks are burned inside the samples .

@autoexecbatman I don't believe that Polyphone soundfont editor is a very perfect software and it may have bugs in its decoder (It gave me a lot of clicks). So I also used OpenMPT to check soundfont and it gave no clicks at all for all samples in SF3 file:

The soundfont is OK, but it contains some loud instruments and MIDI mixer does not correctly work with overloaded sound and simply clips the sound. For comparison (the same game, the same map, all settings are the same except music volume): Volume8:

https://user-images.githubusercontent.com/113276641/226736957-ac061bf3-b091-42a9-a529-4c5fc593cde7.mp4

Volume 10:

https://user-images.githubusercontent.com/113276641/226737188-03269fe4-e334-417b-a1d2-c4a706b3fcec.mp4

Maybe a possible solution for this issue could be to reduce the MIDI volume. (I mean by the engine, not by simple setting volume to 8 or lower value.)

The other solution could be - reduce the volume of samples in SF3 file. :)

[Districh-ru]

After some tests I can consider that the clicks in MIDI playback are produced by SDL midi synthesizer and/or SDL sound mixer. (It makes midi music too loud). I put GeneralUser GS v1.471.sf2 instead of fheroes2.sf3 in the 'files\soundfonts\' folder and got the same clicks at 10 music volume. Then I deleted all soundfonts from fheroes2 to make it use the system midi playback and configured CoolSoft VirtualMIDISynth (which I always use to play midi files) to play sounds from GeneralUser GS v1.471.sf2, started fheroes2 and did not get any clicks in midi music with volume 10.

https://user-images.githubusercontent.com/113276641/226968208-2931eccc-1264-4cd2-b271-692510f08aa8.mp4

@autoexecbatman, could you please test, do you experience clicks in midi music when music volume is set to 8 with original fheroes2.sf3?

[autoexecbatman]

Using: fheroes.sf3(i think it should use this soundfont unless it somehow uses something else). Build: Debug-SDL1. OS: win10 x64 In-game I get no clicks and I hear no clipping at max volume in-game. The music sounds different from yours and my signal is much colder. (got one click due screen recorder)

https://user-images.githubusercontent.com/91700559/227075398-94c61293-8cf7-4954-ae6e-8c85c8f8127e.mp4

... I am curious how another user got clicks in-game ?

• It was converted to *.sf3 using Polyphone if the readme is true.

[autoexecbatman]

https://user-images.githubusercontent.com/91700559/227081127-791b38ca-7b92-4443-8720-9b6543a7f1bb.mp4

*.sf2 files are used in this example here. The same happens actually if I open it with Polyphone using *.sf3 or exporting the samples to *.wav using Polyphone. So the clicks are resolved ? If something clips just turn down the overall gain.

[autoexecbatman]

https://user-images.githubusercontent.com/91700559/227085771-26254755-d381-4360-b32e-d32e0b869623.mp4

[Districh-ru]

Hi @autoexecbatman, many thanks tor testing!

The decoding of .sf3 have some bugs, as we can hear.But in fheroes2 it decodes correctly as we have no clicks with low music volume.

I think that the cause of midi music clipping is the hardcoded 1.2 gain in SDL_mixer:

static void FLUIDSYNTH_SetVolume(void *context, int volume)
{
    FLUIDSYNTH_Music *music = (FLUIDSYNTH_Music *)context;
    /* FluidSynth's default is 0.2. Make 1.2 the maximum. */
    music->volume = volume;
    fluidsynth.fluid_synth_set_gain(music->synth, (float) (volume * 1.2 / MIX_MAX_VOLUME));
}

https://github.com/libsdl-org/SDL_mixer/blob/48608bf6507234c3f434ddbb2931f2ead0df9668/src/codecs/music_fluidsynth.c#L259-L265

As we cannot change this hardcoded value we can reduce the volume for midi playback by 1.2. :)

I'll try to fix this possible midi clipping issue in the next few days.

[oleg-derevenetz]

As we cannot change this hardcoded value we can reduce the volume for midi playback by 1.2. :)

Please note that fheroes2 should (ideally) be backend-agnostic. SDL_Mixer has several MIDI backends, and FluidSynth is just one of them. Even on Windows, if we remove the soundfont file, the native Windows MIDI (with its own issues) will be used. What can we say about builds for other systems, where we have no control at all and do not know which backend is used by SDL_Mixer? We should not change our code to please a particular MIDI backend of several available ones.

As we cannot change this hardcoded value

How about to open a corresponding issue in SDL_Mixer repo and discuss the current behavior with its authors?

[Districh-ru]

Hi @oleg-derevenetz, thanks for warning. Before starting to change our code I wanted to figure out can we determine what backend is used by SDL_mixer. SDL_mixer already has issue https://github.com/libsdl-org/SDL_mixer/issues/284 in which it is mentioned that FLuidSynth midi music is too loud. But for 3 years this issue has not been addressed. :( Maybe because it focuses on changing default playback settings and the loudness of playback is mentioned only in addition. I'll try to open a new issue in SDL_Mixer repo.

Yes, it is not good to make our engine adopt to every backend (especially if we don't know what backend it is). But currently we can wait until SDL_mixer will set FluidSynth gain to 1 and tell all fheroes2 users not to use music volume more than 8 for midi playback. Or try to make a temporary patch if it is possible.

I will not make changes to our engine if there is no correct way to check if FluidSynth is in use.

[Districh-ru]

Also I think we have limit the maximum sound volume to make a room for resampler, as after the resamle some new sample bins can have values more than maximum (or minimum) allowed. In the next example if we increase sample rate by 2 and the resampler dos not have a magnitude limiter than we will have a reading that will be clipped:

This is just an assumption that I want to test soon

[oleg-derevenetz]

But for 3 years this issue has not been addressed. :( Maybe because it focuses on changing default playback settings and the loudness of playback is mentioned only in addition. I'll try to open a new issue in SDL_Mixer repo.

Usually pull requests gain more attention than just issues. If you offer a pull request with the fix, it might be addressed faster. Also in theory we can apply our own patch when building Windows packages in the fheroes2-prebuilt-deps repo (there is already an infrastructure for this), but again - it will work only for our own Windows distribution, but not for 3rd-party builds/packages.

[Districh-ru]

I'll also test if we have this midi clipping issue under Android and Linux

[oleg-derevenetz]

I'll also test if we have this midi clipping issue under Android and Linux

For Android builds there is different backend used by SDL_mixer (Timidity) and on Linux this depends on build settings of SDL_mixer package for the corresponding Linux flavor/distribution. Some may use Timidity and some may use FluidSynth. We have no way to control this.