[FEATURE REQUEST] MIDI MTS SysEx

[OpenSourceAnarchist]

This project is exactly what I was looking for. In short, I'm going to implement the dynamic tuning algorithm proposed in this paper and implemented here. This requires having access to a list of currently playing notes as well as all previously played notes within the previous n seconds, although this is easy enough for me to manually compute with your library. However, I need to set tunings for individual notes, and this is most easily accomplished with SysEx calls in the MIDI Tuning Standard.

See this enum declaration and this API documentation for how Fluidsynth handles this. I can write my own functions to convert between Hz/cents/MIDI frequency data format, but without your library being able to respond to the SysEx calls, I don't know any other way to modify the tunings for all the MIDI notes, for just an octave and extrapolated to the rest, to individual notes, etc. More information about the MTS SysEx calls is available here: http://www.microtonal-synthesis.com/MIDItuning.html

To be honest, I really like the way JZZ implements this as well as the helper functions. See this code and the associated _helper functions as well as the API doc.

I hope this was helpful enough to get an idea on why this is useful and how it could be implemented. Completely understand if this is on the backburner, and I'm open to any ideas you have as well :)

[spessasus]

Hi OpenSourceAnarchist, I knew that midi tuning standard was a thing, but I never quite understood how it works. I'm open to the idea, but I don't clearly understand it even with the helpul links you've provided.

Looking at the fluidsynth API, I don't really understand the "Tuning bank number" and "Tuning program number". Does it mean a tuning for every channel with a specific bank or a program? Or is it just a bank and program offset? And the notes: I understood that it's essentially an array of {note: tuning} in cents, but is this tuning relative or absolute? Since you've mentioned that you can compute the tunings without a problem, I'm assuming that all of the "dump" messages can be omitted? So, to sum it up, you want to support the MIDI_SYSEX_TUNING_NOTE_TUNE and MIDI_SYSEX_TUNING_NOTE_TUNE_BANK?

I'll try to implement that, but without a test MIDI file it will be difficult to do so. Anyways, thanks for the idea. :-)

PS: Here's the current tuning calculation code if that helps.

[spessasus]

I've added MTS octave tuning support. I will work on note tuning support now!

[OpenSourceAnarchist]

I JUST SAW YOUR COMMENT. Thank you so much for working on the MTS octave support. This might not be much help, but I was already writing it as you posted the reply :)

To be honest, my head has been spinning trying to read and remember the MIDI spec, the various additions, fluidsynth API, various js midi library APIs, web midi, and of course they all use similar terms that might be used totally differently in a different part of the spec!

From what I've gathered, MIDI tuning is in cents starting with C. So for standard 12 tone ET, starting with MIDI note 0 (5 octaves below middle C), it would run 0, 100, 200, 300, ..., 12700 since MIDI (defaults?) to 12 TET so the MIDI note # times 100 gives you the cents. It is absolute.

I was under the impression the tuning bank and program numbers are completely independent of MIDI instrument banks or program numbers. I believe it's so performers can switch between a large number of premade tunings on-the-fly. On the fluidsynth dev maling list there is some more discussion on this, you can see they recommend keeping it easy and just modifying tuning bank 0 program 0.

The dump messages may be useful for other people, but for me specifically it wouldn't help. The two SysEx tuning commands you said would be helpful. Honestly you're better off looking at fluidsynth's implementation directly, see all the code for the enum cases here. The MIDI_SYSEX_TUNING_NOTE_TUNE and MIDI_SYSEX_TUNING_NOTE_TUNE_BANK are implemented together.

The most useful things for me are 1) being able to store (or easily retrieve) the most recently played (unmodified!) notes, like "c4" or "midi note 60". I can manually implement this by keeping a running average of how many NoteOn events were seen for each MIDI note in the last X seconds/ticks, or you may have methods for querying active notes at a given time interval through the sequencer. And 2) being able to, in real-time, alter the cents/Hz of the actively voiced notes, including if they are still "On". I was thinking per-note MTS SysEx calls would be the easiest, but after looking at how you calculate tunings, it may be easier to interface directly with your code. I would basically add a function that would override the cents of the note before any of your existing tuning code that takes into account pitch bends on the channel, vibrato, etc. I'm not sure how easy this would be over carefully timed SysEx calls. Both should be straightforward as long as I can easily retrieve all the actively voiced MIDI note numbers.

[spessasus]

Hi, I've added the note tuning support. (MIDI_SYSEX_TUNING_NOTE_TUNE and MIDI_SYSEX_TUNING_NOTE_TUNE_BANK)

Though, I only found one MIDI to test it with (from this issue), so I can't guarantee that it will work every time.

I'm keeping this issue open. If the tuning works fine, please close it. Good luck with your project!

[OpenSourceAnarchist]

I appreciate this a lot truly. I will work on hacking together an implementation of the tuning algorithm with your library and see if any issues or questions come up! I'm so glad I stumbled on your reddit post and repo. Batteries included with MIDI on web is so refreshing <3

[OpenSourceAnarchist]

Just a few questions/comments on the implementation: I'm a little concerned that single note change SysEx calls aren't getting the right priority in the tuning calculation. In src/spessasynth_lib/synthetizer/worklet_system/worklet_methods/voice_control.js, shouldn't the MTS tuning be applied and take full precedence after the more global parameters, e.g. MIDI master tuning, coarse grain, fine tune, scale tunings/channel tunings? I think the MTS spec is ambiguous enough that it's up to you, but I would expect any real-time MTS SysEx messages I send to override any semitone/cents modifications or custom controllers other than the vibrato.

What do you think about something like this instead? In general, can you think of any other interactions or reasons the final pitch might deviate from the MTS frequency?

    // TUNING
    let targetKey = voice.targetKey;

    // calculate tuning
    let cents = voice.modulatedGenerators[generatorTypes.fineTune]
        + channel.customControllers[customControllers.channelTuning]
        + channel.customControllers[customControllers.channelTransposeFine]
        + channel.customControllers[customControllers.masterTuning]
        + channel.channelOctaveTuning[voice.midiNote % 12];
    let semitones = voice.modulatedGenerators[generatorTypes.coarseTune]
        + channel.customControllers[customControllers.channelTuningSemitones];

    // calculate tuning by key
    cents += (targetKey - voice.sample.rootKey) * voice.modulatedGenerators[generatorTypes.scaleTuning];

    // midi tuning standard, overrides other global tuning parameters
    const tuning = this.tunings[channel.preset.program]?.[targetKey];
    if(tuning?.midiNote >= 0)
    {
        cents = tuning.centTuning;
        semitones = 0;
    }
    ...

In general, I just want to make sure no other parts of the library assume 12 tone equal temperament. I'm unsure how this actually plays out with rendering samples, e.g. if there's only one octave and it needs to be pitch shifted. MTS always seemed ad hoc. Oh, and pitch bends are tricky too. Should they take precedence over an MTS tuning? To me, I would expect pitch bends to be overridden by MTS, but I can see an argument for the other case. For my case, I would want to put the logic of handling pitch bends in my own MTS code, e.g. applying a channel pitch bend amount to the cents of the original MIDI note, and then modifying it according to my custom tuning algorithm. If you want pitch bends to take precedence, I can just manually ignore them if my tuning algorithm is being applied. Same logic for master tunings for instruments, e.g. if A4 = 432, I will check and account for this in my own MTS logic for outputting a cents deviation from ET and A4 = 440 Hz.

EDIT: In the MTS Spec, they say:

Recommended Practice (RP-020) Defaults for Scale/Octave Tuning If tuning presets are not supported by the instrument, it is assumed that the initial tuning of the instrument is equal temperament. If presets are supported, it is suggested that the first preset, selected by Bank 0H and Preset 0H, would be equal temperament. Tuning adjusters should begin by selecting Bank 0H, Preset 0H in order to start from equal temperament, if that is the desired behavior.

Perhaps it would be easier to just always read from 0,0 and predefine ET tunings (midi note # * 100), that way you avoid the conditional and can just look up the values for the pitch directly from the bank/program tuning.

[spessasus]

Just a few questions/comments on the implementation: I'm a little concerned that single note change SysEx calls aren't getting the right priority in the tuning calculation. In src/spessasynth_lib/synthetizer/worklet_system/worklet_methods/voice_control.js, shouldn't the MTS tuning be applied and take full precedence after the more global parameters, e.g. MIDI master tuning, coarse grain, fine tune, scale tunings/channel tunings? I think the MTS spec is ambiguous enough that it's up to you, but I would expect any real-time MTS SysEx messages I send to override any semitone/cents modifications or custom controllers other than the vibrato.

What do you think about something like this instead? In general, can you think of any other interactions or reasons the final pitch might deviate from the MTS frequency?

    // TUNING
    let targetKey = voice.targetKey;

    // calculate tuning
    let cents = voice.modulatedGenerators[generatorTypes.fineTune]
        + channel.customControllers[customControllers.channelTuning]
        + channel.customControllers[customControllers.channelTransposeFine]
        + channel.customControllers[customControllers.masterTuning]
        + channel.channelOctaveTuning[voice.midiNote % 12];
    let semitones = voice.modulatedGenerators[generatorTypes.coarseTune]
        + channel.customControllers[customControllers.channelTuningSemitones];

    // calculate tuning by key
    cents += (targetKey - voice.sample.rootKey) * voice.modulatedGenerators[generatorTypes.scaleTuning];

    // midi tuning standard, overrides other global tuning parameters
    const tuning = this.tunings[channel.preset.program]?.[targetKey];
    if(tuning?.midiNote >= 0)
    {
        cents = tuning.centTuning;
        semitones = 0;
    }
    ...

In general, I just want to make sure no other parts of the library assume 12 tone equal temperament. I'm unsure how this actually plays out with rendering samples, e.g. if there's only one octave and it needs to be pitch shifted. MTS always seemed ad hoc. Oh, and pitch bends are tricky too. Should they take precedence over an MTS tuning? To me, I would expect pitch bends to be overridden by MTS, but I can see an argument for the other case. For my case, I would want to put the logic of handling pitch bends in my own MTS code, e.g. applying a channel pitch bend amount to the cents of the original MIDI note, and then modifying it according to my custom tuning algorithm. If you want pitch bends to take precedence, I can just manually ignore them if my tuning algorithm is being applied.

EDIT: In the MTS Spec, they say:

Recommended Practice (RP-020) Defaults for Scale/Octave Tuning If tuning presets are not supported by the instrument, it is assumed that the initial tuning of the instrument is equal temperament. If presets are supported, it is suggested that the first preset, selected by Bank 0H and Preset 0H, would be equal temperament. Tuning adjusters should begin by selecting Bank 0H, Preset 0H in order to start from equal temperament, if that is the desired behavior.

Perhaps it would be easier to just always read from 0,0 and predefine ET tunings (midi note # * 100), that way you avoid the conditional and can just look up the values for the pitch directly from the bank/program tuning.

I implemented mts the same way that TiMidity++ does (as fluidsynth doesn't seem to work because of that bug i got the mts.mid from)

Your code wouldn't work at all, because: take a look at this The this.tunings table has two properties: midiNote and centTuning.

• The midiNote is like "coarse tune". It shows the base MIDI note to use (the first byte of the MTS frequency).
• Cent tuning is just the fine tune addition to that note being used, maxing out at 100 and min -100. The semitones being forced to 0 cause the midiNote from tuning to be ignored, causing the note to have the incorrect tuning.

Not to mention that discarding the tunings is catastrophic because:

1. We discard soundfont's tuning generators which is a big no-no. These ALWAYS have to apply. It's not controllable by MIDI and it ensures that the sample plays correctly.
2. Every sample has a specific "original key" number which is used to determine the proper pitch (this line: (targetKey - voice.sample.rootKey) * voice.modulatedGenerators[generatorTypes.scaleTuning];) Setting semitones to 0 will cause the sample to sound like it's directly stored in the file. This effectively kills ALL tuning (including MTS) completely. Currently, MTS overrides the "original key" which tunes the sample to the desired frequency. But setting semitones to 0 overrides that, and everything else.

All of the tuning controllers (except for the soundfont ones) are always initialized to 0 by default:

• If you want to avoid pitch wheel, simply set pitch bend range to 0 via RPN.
• If you want to avoid coarseTune and fineTune RPNs, simply don't set them. They are always set to 0 by default.
• You can't avoid vibrato, as it's the part of the sf2 standard. You can set the modulation wheel to 0, but if the soundfont has vibrato hard-coded, it will be on. But most of them just use the mod wheel so that solution would work.
• Custom controllers are only applied with the synth's functions (except for the master tuning which is for the MIDI sysex I think), and they are always set to 0 by default. Simply don't change them and everything should work as expected.

As I said earlier, I only had one MTS file to test stuff with. If you have one which doesn't work correctly, send it here and I'll look into it.

[OpenSourceAnarchist]

Yes, your implementation and precedence/ordering make more sense now. I didn't understand the purpose of midiNote in this.tunings. I thought midiNote was referring to the original midi note number, and that centTuning was the total deviation from the standard ET pitch in cents. Of course it is obvious when looking at const tuning = this.tunings[channel.preset.program]?.[targetKey];, sorry for the confusion :)

I believe I understand the chain now, thank you again! I'll definitely learn with some edge cases and sanity checking the final pitches. But this seems much simpler and I can handle all the different situations in my own logic.

[spessasus]

Hi @OpenSourceAnarchist, How is the tuning algorithm going? This issue has been open for around a month now, so if the MTS works fine, I'll close it. Let me know!

[OpenSourceAnarchist]

I'm not done with it, but if I find any issues I'll reopen this. Otherwise I'll just let you know when it's done so you can check it out! I just moved so I've been adjusting to new routines and left the last part unfinished :( Thanks for implementing this again, it means a lot!