BSE: SF2: use shared engine module instead of global soundfont lock

[swesterfeld]

This moves SoundFont processing to a common engine module which is connected to all sound font osc modules. This ensures that the data is ready when the modules are called, and process_fluid_L is only called once in the common module. The global soundfont lock problem should be solved by this PR.

Note that the sound_font_osc_process function still acquires the lock for a brief amount of time, but no audio generation takes place while the lock is held.

[swesterfeld]

Recently, we discussed this merge request, and you (Tim) said you do not want to merge code that locks a mutex during audio processing due to priority inversion.

So, what does the lock protect here? There is one global fluid_synth instance that is used by both, the soundfont repo and the soundfont osc code. There is also a bunch of data shared between the soundfont osc modules that is protected by the lock. Locks are acquired

• by the soundfont repo when loading/unloading soundfonts
• by the soundfont osc modules when rendering audio

Like the wave repo code, we never load/unload soundfonts if the repo is prepared, so effectively the soundfont repo will not lock the global mutex while the project is playing.

The soundfont osc modules are

• one common module that does the actual rendering (1)
• per osc modules that just copy the data out, and maybe change the preset of the current voice (2)

But these modules are all executed within the engine process() callback so they effectively run with the highest priority we have. So there is no priority inversion possible here. Also note that while (1) takes a while because it renders the data, (2) is really cheap, because the audio data is already available.

[tim-janik]

Recently, we discussed this merge request, and you (Tim) said you do not want to merge code that locks a mutex during audio processing due to priority inversion.

Engine modules must generally strive to finish their work in a predictable time period (soft realtime rendering) and not block for arbitrarily long periods. Examples that have to be avoided are:

• Acquiring a mutex, since wait times for locks are not predictable.
• Files must not be accessed from engine modules as this can also take arbitrarily long.
• Filter table calculations (like approximation methods) also have to be moved out of engine modules.
• Priority inversion (engine modules blocking on a lower priority thread) must be avoided;
• Also, at the CPU / memory-bus level synchronization primitives are very expensive, so use of atomics/barriers should be avoided in engine modules if at all possible.
• Also, mutexes and conditions are hard to reason about especially in an already complicated highly multi-threaded context like the engine. So other primitives like lock-free message queues should be used instead.

So priority inversion is just one of them.

Given the above set of limitations, implementing engine modules would be very hard. That's why we added engine-jobs that can be queued and processed (in a lock-free manner) on engine modules. The way that works is as follows:

• Any state needed by an engine module is loaded/generated/computed in the BSE thread and wrapped by a callback into a closure that is enqueued on the corresponding engine module(s).
• Before module processing, the jobs are executed in engine context and can update the module state as needed (without causing further allocations or using mutexes).
• After job execution, the closure context is destroyed/deleted in the BSE thread.

So state that might ordinarily be synchronized between threads via a mutex should in the context of an engine module be reference-counted or copied and transferred into the engine module via a callback closure, without needing additional synchronization primitives.

So, what does the lock protect here? There is one global fluid_synth instance that is used by both, the soundfont repo and the soundfont osc code. There is also a bunch of data shared between the soundfont osc modules that is protected by the lock. Locks are acquired

* by the soundfont repo when loading/unloading soundfonts

* by the soundfont osc modules when rendering audio

Like the wave repo code, we never load/unload soundfonts if the repo is prepared, so effectively the soundfont repo will not lock the global mutex while the project is playing.

The soundfont osc modules are

* one common module that does the actual rendering (1)

* per osc modules that just copy the data out, and maybe change the preset of the current voice (2)

But these modules are all executed within the engine process() callback so they effectively run with the highest priority we have. So there is no priority inversion possible here. Also note that while (1) takes a while because it renders the data, (2) is really cheap, because the audio data is already available.

Why exactly is a mutex needed in the first place, let alone in an engine module? As stated above, ultimately the code needs to be reworked to avoid the lock and use jobs or similar means to update engine module state. Just to make sure that's understood, it is not ok for engine modules to be blocked while file IO ocours in another thread (like loading a soundfont).

[swesterfeld]

Why exactly is a mutex needed in the first place, let alone in an engine module?

Because multiple engine modules share state, especially because multiple engine module access the same fluid_synth_t instance, and could (possibly) call API functions on the same fluid synth instance at the same time.

As stated above, ultimately the code needs to be reworked to avoid the lock and use jobs or similar means to update engine module state.

I agree that ultimately the solution I submitted is not perfect.

To avoid long discussions, I'll try to put more work into this branch and resubmit a new version that can hopefully do this without lock.

[tim-janik]

Why exactly is a mutex needed in the first place, let alone in an engine module?

Because multiple engine modules share state, especially because multiple engine module access the same fluid_synth_t instance, and could (possibly) call API functions on the same fluid synth instance at the same time.

In that case, there should be one engine module around the fluid_synth_t instance, and the other modules should connect to this one module as input.

As stated above, ultimately the code needs to be reworked to avoid the lock and use jobs or similar means to update engine module state.

Can we get rid of the shared state once the fixed fluidsynth 2.0.5 is out, or is this unrelated?

[swesterfeld]

In that case, there should be one engine module around the fluid_synth_t instance, and the other modules should connect to this one module as input.

Yes, this PR does about half of what you ask; it uses one shared engine module around the fluid_synth_t instance (good), but on the other hand accesses the fluid_synth_t instance from all engine modules (bad), so it still needs a lock. It could be fixed in this PR, but I now believe it is better to use a different approach; on fluid_synth_t instance per osc, as submitted in https://github.com/tim-janik/beast/pull/102

As stated above, ultimately the code needs to be reworked to avoid the lock and use jobs or similar means to update engine module state.

Can we get rid of the shared state once the fixed fluidsynth 2.0.5 is out, or is this unrelated?

This is unrelated. Using fluidsynth 2.0.5 mainly allows us to fix the deprecation warning.

[tim-janik]

Stefan, can we close this in favor of #102 ?

[swesterfeld]

Stefan, can we close this in favor of #102 ?

Yes, I'm closing this PR now.