Voice-Allocation Bug When Using Two or More Simultaneous Samples in a Preset

[derselbst]

Originally created by: Mr_Bumpy

Under certain circumstances, when out of polyphony, FluidSynth seems to steal newly-created voices if more than one sample is triggered at the same time. This file contains a test SoundFont and audio examples, as well as more details on the bug.

From the included documentation:

---

I have discovered an apparent voice-allocation bug that reveals itself under the following circumstances:

1. more than one sample is played with each keypress
2. the preset's volume envelope has long release values
3. the sustain pedal is not being held down
4. enough notes are being played to exhaust Fluidsynth's available polyphony (which causes FluidSynth to choose which existing voices get the axe to make room for the new ones)

What seems to happen is that when FluidSynth runs out of voices and a new note is played, it has to choose an existing note to kill as determined by FluidSynth's voice-stealing logic. Unfortunately, if a note is played that requires more than one voice, FluidSynth seems to steal the first newly allocated voice to give it to the second.

Another way of looking at it is:

1. FluidSynth is out of notes, and receives an event to trigger a new note.
2. The new note requires two voices of polyphony.
3. FluidSynth steals an older voice so that the sample 1 of the new note can play.
4. FluidSynth steals another voice so that the sample 2 of the new note can play, except that it steals the voice that was just assigned to sample 1 instead, so sample 1 is cut short before it even gets started.

For the final MIDI example, I created a piano preset that uses two samples per key, one panned hard left, and the other hard right and features a long volume envelope release time. The example plays a fast glissando, which causes FluidSynth to run out of polyphony. When that happens, you only hear the left sample playing on new notes, because the right sample is never getting a chance to sound.

Reported by: *anonymous

Original Ticket: fluidsynth/tickets/27

[derselbst]

Original comment author: berarma

This comment is found in function fluid_defpreset_noteon, but I don't know which code is referring to:

/* Store the ID of the first voice that was created by this noteon event.

• Exclusive class may only terminate older voices.
• That avoids killing voices, which have just been created.
• (a noteon event can create several voice processes with the same exclusive
• class - for example when using stereo samples) */

Original comment by: elementgreen

[derselbst]

Original comment author: diwic

This patch fixes the actual case, but I'm not sure that it is the best solution. The theory is, if a voice is during its delay or attack section, it should be judged from it's coming max volume instead of its current volume.

--- src/fluid_synth.c (revision 191)
+++ src/fluid_synth.c (arbetskopia)
@@ -2170,21 +2170,29 @@

this_voice_prio -= (synth->noteid - fluid_voice_get_id(voice));

/ take a rough estimate of loudness into account. Louder voices are more important. /

- if (voice->volenv_section != FLUID_VOICE_ENVATTACK){
+ if (voice->volenv_section > FLUID_VOICE_ENVATTACK){

this_voice_prio += voice->volenv_val * 1000.;

+ } else {
+ this_voice_prio += voice->volenv_data[FLUID_VOICE_ENVATTACK].max * 1000.;

}

/ check if this voice has less priority than the previous candidate. /

Original comment by: elementgreen

[derselbst]

Original comment author: diwic

Here's the diff a little more readable:

--- src/fluid_synth.c   (revision 191)
+++ src/fluid_synth.c   (arbetskopia)
@@ -2170,21 +2170,29 @@
     this_voice_prio -= (synth->noteid - fluid_voice_get_id(voice));

     /* take a rough estimate of loudness into account. Louder voices are more important. */
-    if (voice->volenv_section != FLUID_VOICE_ENVATTACK){
+    if (voice->volenv_section > FLUID_VOICE_ENVATTACK){
       this_voice_prio += voice->volenv_val * 1000.;
+    } else {
+      this_voice_prio += voice->volenv_data[FLUID_VOICE_ENVATTACK].max * 1000.;
     }

     /* check if this voice has less priority than the previous candidate. */

Original comment by: elementgreen

[derselbst]

Original comment author: Mr_Bumpy

This only solves half the problem. Let's say I play a bunch of samples that are stereo, until I run out of polyphony. Then I play a single mono sample. FluidSynth will kill a single voice - say, the left channel of my oldest sound - and give it to the new sound. So, the left channel has dropped out, but not the right. FluidSynth should modified so that when it steals a voice, it kills all other voices that are associated with the same note on. This could be done by assigning every note a note-on event number, and then when we steal a note, we kill any voices with the same note on number as the voice we're stealing.

Original comment by: elementgreen

[derselbst]

Original comment author: diwic

In some cases perhaps killing all voices belonging to the same note-on makes sense, but I think that in other cases it would make more sense to have it as it is. If a preset contains more than one voice per note, often one voice is the "major" one, and the others are much lower in volume. Given that case, I think I would prefer killing all the minor voices first, before killing the major ones.

Original comment by: elementgreen

[derselbst]

Original comment author: Mr_Bumpy

Diwic, your logic makes sense. Perhaps stereo samples could be detected and handled as a single sample. That would be the best of both worlds. Then minor voices could be killed off first, but when a stereo sample is detected, both samples for it are killed.

Original comment by: elementgreen

[derselbst]

This is really 2 separate tasks. The first being the addition of sample groups, which are triggered atomically, for stereo samples for example. Currently FluidSynth may trigger a sample pair in different audio buffers, which could mean that they are delayed by 64 samples. This is contrary to the SoundFont specification. Adding sample groups would allow the group to be started atomically and allow for voice stealing logic to take advantage of that feature also. Added another ticket for this: #46

There was some discussion a number of years ago about improving the voice stealing algorithm, with configurable settings parameters for controlling the voice stealing behavior. We should probably review this, since I think we arrived at a fairly good scheme for voice stealing, but it has as of yet not been implemented.

http://lists.gnu.org/archive/html/fluid-dev/2005-10/msg00017.html

Original comment by: elementgreen

[derselbst]

Original comment author: Mr_Bumpy

Okay, I have noticed that when using a patch with multiple samples-per-key and a long release, FluidSynth's polyphony gets chewed up very quickly. Here's a link to a test SoundFont to demonstrate this: http://www.schristiancollins.com/temp/voice-stealing-test.sf2. The test SoundFont features five samples-per-key. When FluidSynth is out of voices, only one of the five samples sound for newly played keys (and sometimes I can't even hear one). In my opinion, newly played notes should always sound, so I propose adding a new rule in the voice-stealing scheme whereby notes still in the attack phase are given a higher priority. This seems to solve the problem with the test case.

Here is the code a friend of mine cooked up (using 1.0.9):

In fluid_synth.c:
AFTER THIS:

    } else if (_RELEASED(voice)){
      /* The key for this voice has been released. Consider it much less important
       * than a voice, which is still held.
       */
      this_voice_prio -= 2000.;
    }

ADD THIS:

    if ( _ATTACK(voice))
    {
      // This note is just starting..  better to kill a voice later on in it's cycle
      this_voice_prio += 1000;
    }

In fluid_voice.h:
AFTER THIS:

#define _ON(voice)  ((voice)->status == FLUID_VOICE_ON && (voice)->volenv_section < FLUID_VOICE_ENVRELEASE)

ADD THIS:

#define _ATTACK(voice)  ((voice)->status == FLUID_VOICE_ON && (voice)->volenv_section < FLUID_VOICE_ENVHOLD)

Original comment by: elementgreen

[derselbst]

If this is the general improve-voice-stealing bug, it makes sense to add the text I found in http://www.midi.org/techspecs/gmguide2.pdf about this:

GM Voice Allocation - Overflow and Channel Priority
• Allocation priority should always be given to the most recent voice(s) played. Second
priority should be given to the loudest voice(s) currently being played. In addition,
manufacturers should implement other musically-oriented solutions, such as stealing
individual oscillators from multiple-oscillator programs, and reassigning oscillators
which can no longer contribute substantially to the perception of a note.
• Notes on certain MIDI Channels should have priority over others. That is, Channel 10
receives highest priority, followed by Channel 1, then Channel 2, etc., with Channel 16
receiving lowest priority.
Composer Recommendations: It is advised that MIDI file authors insure that voice
overflow situations (where more than 24 notes need to be played simultaneously) are
avoided. A MIDI file checker application can help identify these situations.
Details:
Description of issue:
Roland and Yamaha recommend specific Channel assignments for specific
instrumentation, supposedly to provide better compatibility during a voice-overflow
situation. If allocation schemes can contribute to compatibility problems, is there a
recommendation for how notes (and Channels) should be prioritized?
Findings:
The respondents were split evenly between last-note and highest-volume priority.
Three used various combinations of priorities. The issue of Channel priority was not
addressed in the survey; however, many manufacturers including Roland and
Yamaha follow the scheme described above, and this is also the scheme recommended
by the IASIG for the upcoming Downloadable Sounds specification for synthesizers.

Original comment by: diwic

[derselbst]

• owner changed from jgreen to diwic3
• status set to accepted

Original comment by: diwic

[derselbst]

• status changed from accepted to closed
• resolution set to fixed

I believe this was fixed in 1.1.2 with the new overflow settings.

Original comment by: diwic