Select soundfont samples by frequency instead of midi note numbers

[Naturseptime]

I have problems using FluidSynth together with custom tunings for microtonal music. The problem is that the Soundfont samples are selected by the midi-notes instead of the tuned frequency.

Steps to reproduce: 1) Use a soundfont with different samples for different pitches (I tested with the Unison Soundfont) 2) We tune three notes to the same frequency.

settuning 0 0 0
tune 0 0 60 6000
tune 0 0 48 6000
tune 0 0 36 6000

noteon 0 60 100
noteon 0 48 100
noteon 0 36 100

Each of the notes sounds very different because distinct Soundfont samples were used.

This is a rather big issue when tunings with completely different scales are used (i.e. for microtonal music). Always approximating pitches by the nearest midi semitone would be an ugly workaround here because for fine-grained scales multiple pitches would map to the same midi note and a lot of dynamic retuning would be neccessary inbetween.

Maybe one could add an option to enable sample selection by tuning frequencies instead of midi note numbers. Thanks!

[mawe42]

I looked into this recently as well (https://github.com/FluidSynth/fluidsynth/issues/102#issuecomment-801508582). And yes, I think having an option to change the behaviour would be great.

Would you be willing and able to draft a PR for this?

[mawe42]

Looking into this a little more, I think it would be really difficult to implement properly. The easiest solution might be to calculate a different MIDI key based on the channel tuning before executing the note-on in the soundfont loader. But that approach has a lot of drawbacks which would probably make this feature useless. Like two notes with very similar tuning which would map to the same base note, effectively cancelling each other out (because a note-on on the same channel with the same key stops the previous voice started by that key).

So would have to keep track of the actual key and the "tuned" key for each note-on and make sure that we use the correct one in each of the many places where currently only the key and velocity of the note is relevant. This will not be trivial... and I think it would have to touch a lot of function signatures, some of which are exposed via the public API.

[Naturseptime]

Would it be possible to use the tuned key only for the soundfont sample selection and keep the actual key for all other purposes?

Adding the following lines to fluid_defpreset_noteon in src/sfloader/fluid_defsfont.c

fluid_tuning_t *tuning = fluid_channel_get_tuning(synth->channel[chan]);
int tuned_key = tuning ? (int)(fluid_tuning_get_pitch(tuning, key) / 100.0 + 0.5) : key;

and replacing key by tuned_key in the two subsequent calls to fluid_zone_inside_range seem to work very well.

If this solution is feasible I would make a PR.

[mawe42]

That is a possible solution, it's basically what I mentioned in the first paragraph above. But like I said, it does have some drawbacks. If you would tune three keys to 440, 443 and 449 Herz, they would all map to note number 69 and would be treated like the same note. Which in turn means: if you send a chord of these three notes, you would only hear a single note (the last one sent) because each new note-on would stop the previously started voice on the same key/channel.

[mawe42]

it does have some drawbacks [...] they would all map to note number 69

Hm... I think you're right, simply replacing the key argument to fluid_zone_inside_range is probably enough and doesn't have the mentioned drawbacks. Cool, maybe it's much simpler than I thought!

A PR would be great, then we can test it. What do you think @derselbst ?

[derselbst]

I'm so far away from microtonal tuning, I have no clue. If you guys have a plan, let's do it. The only question I have: Should we add a new setting which enables this magic-key-replacement-sorcery? If you say that this is only natural sounding way for any possible tuning and that the old behavior simply sucks and nobody expects it, then I could live without such a setting.

[mawe42]

If you say that this is only natural sounding way for any possible tuning and that the old behavior simply sucks and nobody expects it, then I could live without such a setting.

Not sure... I think there are valid use-cases for both behaviours. So I would vote for a setting and default to the current behaviour.

[Naturseptime]

I agree with @mawe42 that there are valid use cases for both behaviours:

a) Every key plays a different instrument (e.g. in percussion sets). In this case we definitely want to keep the old behaviour where the sample is selected by the key number (logically it wouldn't make much sense to detune a drum into a cymbal)

b) All keys play the same instrument and the soundfont samples differ only by slight variations of pitch-dependent timbres (e.g. pianos where the lower notes have more overtones than higher notes). In this case it is more intuitive to select the sample by the tuning pitch. If a user says "Play a piano on 1000 hz for me" he would expect that the synthesizer figures out the best matching sample by itself.

Therefore I vote for an option and it should be configurable for each channel. API suggestion:

int fluid_synth_set_sample_selection_mode(fluid_synth_t *synth, int chan, int mode);
int fluid_synth_get_sample_selection_mode(fluid_synth_t *synth, int chan);
FLUID_CHANNEL_SAMPLE_SELECTION_BY_KEY = 0
FLUID_CHANNEL_SAMPLE_SELECTION_BY_PITCH = 1

[mawe42]

I like the API suggestion. And I think we should also make this feature available to command-line users. So I would propose a new (real-time) option: synth.sample-by-pitch-channels. Argument would be a comma-separated list of channel numbers, each channel in the list gets set to selection-by-pitch mode. Default is an empty list, which means that all channels are in selection-by-key mode.

And one more thought: I think we should also take channel coarse and fine tune into account when calculating the closest MIDI note for a pitch. And if that might be problematic in some cases, then I would suggest three modes:

FLUID_CHANNEL_SAMPLE_SELECTION_BY_KEY = 0
FLUID_CHANNEL_SAMPLE_SELECTION_BY_TUNING = 1
FLUID_CHANNEL_SAMPLE_SELECTION_BY_PITCH = 2

BY_TUNING would only consider the channel tuning, BY_PITCH would also take coarse and fine tune into account. Although then we would also need to use a different way to configure this via command-line settings. Maybe two lists: synth.sample-by-tuning-channels and synth.sample-by-pitch-channels.

[derselbst]

Every key plays a different instrument (e.g. in percussion sets).

The magic-key-replacement-sorcery would only apply, if a tuning has been set. So, would it make any sense to activate a microtonal tuning on a percussion set? I cannot imagine. So - unless there is another use-case for "Every key plays a different instrument" that I'm currently missing, I do wonder why we would need FLUID_CHANNEL_SAMPLE_SELECTION_BY_KEY at all?

Thinking this even further: If we ever decide to apply a neutral tuning to all channels by default (as recently suggested in #102), one could still apply the FLUID_CHANNEL_SAMPLE_SELECTION_BY_KEY behavior depending on whether a channel is melodic or drum.

BY_TUNING would only consider the channel tuning, BY_PITCH would also take coarse and fine tune into account.

If there is any chance to keep it simple, I would prefer to keep it simple, i.e. non-configurable. I admit, I have no clue musical background and whether it makes sense to decide for only one strategy. I'm just thinking of a theremin or any brass instruments where different notes are played, along with MIDI pitch applied to achieve some glissando effect. And how it would sound like with the magic-key-replacement-sorcery in both cases (BY_PITCH and BY_TUNING).

[Naturseptime]

The magic-key-replacement-sorcery would only apply, if a tuning has been set. So, would it make any sense to activate a microtonal tuning on a percussion set? I cannot imagine.

The only usecase I can imagine is that people want to tune one of their drums to get a brighter or darker timbre. For such cases our "magic" should not apply there (as it wouldn't make any sense for drums anyway).

Thinking this even further: If we ever decide to apply a neutral tuning to all channels by default (as recently suggested in #102), one could still apply the FLUID_CHANNEL_SAMPLE_SELECTION_BY_KEY behavior depending on whether a channel is melodic or drum.

I think using always BY_PITCH for melodic channels and BY_KEY for drum channels gives a very good default setup which only barely needs further adjustment. The only disadvantage I see is that the drum mode locks you into the drum instrument bank and maybe a few people want use BY_KEY also for melodic instruments (but I hardly can find a usecase for it)

On-the-fly-retuning/glissando: I think we should always use channel fine/coarse tuning for sample selection too (for simplicity). The question is what to do when notes are retuned during playing. The only simple way I see is to keep the chosen Soundfont sample/preset from the beginning until the note ends (as dynamic morphing between multiple samples is hard)

[derselbst]

The only usecase I can imagine is that people want to tune one of their drums to get a brighter or darker timbre

But one could simply use the regular pitch bend for brighter and darker timbres. I don't think one would ever need the frequency-precise tuning of a drum as provided by the microtonal tuning.

The only disadvantage I see is that the drum mode locks you into the drum instrument bank

This behavior actually strikes me as well. But I think this could be solved separately if need be.

maybe a few people want use BY_KEY also for melodic instruments (but I hardly can find a usecase for it)

If somebody has such a use-case, we could still make it configurable. For now I would say: We ain't gonna need it.

I think we should always use channel fine/coarse tuning for sample selection too

Ok, let's try it.

The only simple way I see is to keep the chosen Soundfont sample/preset from the beginning until the note ends

Yes, definitely keep it until note off.

[mawe42]

I can also only think of the the drum selection as a use-case for sample-by-key. And even though we haven't had many people filing issues or starting discussions on the tuning feature: if they have, then the fact that FluidSynth doesn't select samples by frequency was almost always a complaint. So making this the new default for melody channels and sample-by-key for drum channels sounds good to me. My main motivator for suggesting a setting was that we change existing behaviour that has been around for a very long time. But I guess we could always wait for the complaints to see if anybody cares at all :-)

And yes, always also considering channel coarse/fine tune when a tuning is set on the channel sounds like a good idea. And that might even be a reason not to implement default tuning for each channel, as I suggested in #102. Because we now have the following options:

1. no channel tuning (default): normal behaviour, sample selection by key
2. set (empty) tuning on a channel: sample selection by frequency (effectively base freq of the key + coarse/fine tune of the channel)
3. alter tuning on a channel: sample selection by modified frequency + coarse/fine tune

I think I might even have a use-case for option 2. :)

And I also agree: sample selection should only happen at note-on.

Oh and please note: glissando or short-time pitch effects are usually done with the channel_pitch_bend, not the channel coarse/fine tune RPNs. Pitch bend should not be considered when selecting a sample-by-pitch.

[mawe42]

Side note: I wouldn't consider this new behaviour sorcery or magic at all. I think it's a really good and simple solution to make our tuning support musically useful.