How do I merge multiple SoundFonts into one?

[LiHongyao]

Hello, how do I get a soundFontBuffer from a Soundfont2 instance? Because I need to load multiple sound library, I use SoundFont2 MergeSoundfonts () method. When creating an instance of Synthesizer, the second parameter requires the soundFontBuffer type.

[spessasus]

Hi, For now, you can use the write method, but it's quite slow as it writes the entire soundfont binary. It will work, but it's slow.

const buffer = yourMergedSoundfont.write().buffer:

I can add a new method to the synthesizer called addAdditionalSoundFont which will merge the soundfonts in the worklet thread, without writing them which should be a lot faster.

[LiHongyao]

Thank you very much and look forward to addAdditionalSoundFont's release！

[spessasus]

Hi, I have added the soundfont manager! Check the docs. You can stack multiple soundfonts within the synthesizer and it will use them all. I hope you like it!

[LiHongyao]

Hello, I may not know how to use SoundfontManager, or I may not know when to use SoundfontManager. I can give you a general description of my usage scenario.

I have 92 timbre, so there are 92 files with the suffix sf3 in the project. Now I am developing the music score function module, if 92 files are loaded at one time, it will be super slow. Therefore, my current solution is that when users select style templates, each template will have a corresponding set of required timbre ids, such as style template 1. The sound color assumption he needs is [1, 2, 3, 5, 8]. After the user selects the corresponding style template, I will determine whether the current Soundfont2 instance exists. If it does not exist, I will load 1.sf3, 2.sf3, 3.sf3... It then calls MergeSoundfonts to combine them together, and finally updates the Synthesizer instance. If so, I'll get the presets attribute from the current Soundfont2 instance, and then iterate through it to get the element's program attribute, compare it to the style template ID [1, 2, 3, 5, 8], filter the sound files that haven't been loaded yet. Assuming that the filter [3, 8] is not loaded yet, I will load 3.sf3 and 8.sf3 and update the instance of Synthesizer.

The following code is the version I used a long time ago, you can take a look:

I update Synthesizer every time I call loadSf3, and the code looks like this:

So, how do I adapt the code to the latest version?

Finally, thank you very much for your patience in reading my questions. I have been recommending the library you maintain to my friends around me, and I have been saying that you are a patient and responsible person, because every time I ask questions, you are patient in answering them.

[spessasus]

Hi, first of all, thank you for recommending my library to your friends! Seeing more people using my work motivates me to delevop it further :-)

Secondly, if each sf3 contains only one preset, I highly recommend merging them into one file with tools like Polyphone. That way you have one large soundfont with mutiple sounds assigned to each program (like the soundfont used in the demo). Then you can simply use synth.programChange(channel, program) to switch timbres which will be super fast since not reloading is required.

But if you want to keep the files separate, here's how I'd adapt your code: Firstly, avoid creating the new synthesizer instance unless absolutely necessary. To reload it with a new soundfont, do

await synth.soundfontManager.reloadManager(soundfontBuffer);

but that method only allows one soundfont on the soundfont stack.

To take the full advantage of the manager, you can use the soundfont stack as follows (I am omitting the bank offset as it doesn't seem to be useful in your case):

1. When you first initialize the stack, the main soundfont (the one you've passed to the synthesizer has the id main).
2. You can see the stack by using synth.soundfontManager.soundfontList (below is the browser's console output):

   
   synth.soundfontManager.soundfontList
   Array [ {…} ]
   ​
   0: Object { id: "main", bankOffset: 0 }
   ​
   length: 1
   ​
   <prototype>: Array []

for example, assuming you have your list of soundfonts like in the first example you've provided:

```ts
const sf3Buffers: ArrayBuffer[] = await Promise.all((sf3Urls.map(fetchSf3)));

// initialize the synth with the first buffer,
// get the first buffer and delete it from the list
const firstBuffer: ArrayBuffer = sf3Buffers.slice(0, 1)[0];
const synth: Synthetizer = new Synthetizer(audioCtx.destination, firstBuffer);
await synth.isReady;

// add all other buffers
for(let index: number = 0; index < sf3Buffers.length; index++)
{
    const buf: ArrayBuffer = sf3Buffers[index];
    const newID: string = `timbre ${index}`;
    await synth.soundfontManager.addNewSoundFont(buf, newID);
}

// check the new soundfont list
console.log(synth.soundfontManager.soundfontList);

// rearrange the buffers if needed
synth.soundfontManager.rearrangeSoundFonts(["timbre0", "main", "timbre1" /*etc...*/])

I hope this helps. If it does, feel free to close this issue. If it doesn't, let me know!

[LiHongyao]

I am very excited to receive your reply, thanks again!

The use of Polyphone you mentioned does not show much for the current requirements, because the merged sf3 file is about 300 megabytes, and I tested that it would take more than two minutes to load in the production environment, so I decided to use the form of on-demand loading.

I just finished writing the code according to your example, but I encountered an exception related to the file, which I need to communicate with my colleague who specializes in dealing with sf3 files tomorrow morning, so I will give you a reply tomorrow morning.

Because I am in China, there will be jet lag, I will go to bed, tomorrow 7:00 up running, good night brother! If you have a chance, welcome to Sichuan, China, please eat hot pot.

[spessasus]

Hi, The problem is you've saved the file as sf2 with Polyhone. This means uncompressed samples. Open the file, then in the top right corner choose "Export soundfonts" then choose format "sf3". This will shrink the file size a lot!

For example, the SGM soundfont in my demo weighs around 144MB in the sf2 format while it weighs only 20MB in the sf3 format.

I hope this helps!

PS: If you want to read more about the sf3 format, see this: https://github.com/FluidSynth/fluidsynth/wiki/SoundFont3Format

[LiHongyao]

Hi, Based on the example you provided, the running program console throws an exception as follows： 1） spessasynth_lib.js?v=831eda4f:911 Error: SyntaxError: Invalid chunk header! Expected "riff" got "" at Synthetizer.handleMessage (spessasynth_lib.js?v=831eda4f:3537:25) at Synthetizer.worklet.port.onmessage (spessasynth_lib.js?v=831eda4f:3442:47) 2） Uncaught (in promise) DOMException: Failed to execute 'decodeAudioData' on 'BaseAudioContext': Unable to decode audio data

Attached is my code file.

App.tsx.zip

Well，Please allow me to introduce my application scenario to you again. We are making a demand for an AI music classroom. Users can choose style templates to generate harmony and accompaniment, because our timbre library is super large, and now it has been converted into a file in sf3 format, which used to be an independent file with about 300 MB. Loading sf3 resources is particularly time-consuming, which is expected to take more than two minutes, so we plan to load on demand, that is to say, the timbre library will be divided into multiple files according to timbre. Currently, there are 92 files in sf3 format. When users select a template, each template will have a field to identify the timbre required by this template. At the same time, the user can change the musical instrument. When the user selects a musical instrument, because the corresponding sound library has not been loaded, I also need to load and merge the new musical instrument's corresponding sound file into the previous loaded sound library resource. In order to optimize the performance, whether it is to switch templates or replace instruments, I will judge whether the corresponding timbre file has been loaded at present, if it has not been loaded, I will load it, if it has been loaded, I will not do any processing.

I hope you can give me some advice, and thank you for taking the time to read my messages to you, I sometimes feel stupid, thank you for your patience to answer my questions, I wish you a happy life!

[spessasus]

Hi again, Looking at your code and solution, it seems that loading all timbres at once is not a good idea, you're right.

So, here's the code (stripped of react stuff) I wrote for you, that hopefully meets your requirements.

First of all, I've removed the creation of multiple synths and sequencers. I can already tell you that:

1. Due to the way sequencer is programmed, only one sequencer can be used with the synthesizer.
2. One synthesizer is probably enough for most use cases.

This code only loads the first timbre, and you should call loadNewTimbre to load a new timbre and merge it into the synth. That way only the first timbre is loaded and if user loads a new one, it will fetch it and merge it within the syntheszier. I hope that helps!

import { Sequencer, Synthetizer } from "spessasynth_lib";

// cache timbre data
const cachedTimbres: {[key: string]: ArrayBuffer} = {};

// create context (it will suspend, we can resume it later)
const context = new AudioContext({
    sampleRate: 44100,
    latencyHint: "interactive"
});
// add worklet
await context.audioWorklet.addModule("/worklet_processor.min.js");

// load the first timbre as a base
const timbre: ArrayBuffer = await (await fetch(`/soundfonts/the_first_timbre.sf3`)).arrayBuffer();
// cache it
cachedTimbres["the_first_timbre"] = timbre;

// create synth and initialize seq as undefined
const synth: Synthetizer = new Synthetizer(context.destination, timbre);
await synth.isReady;
let seq: Sequencer;

// events
const onPlay = () => {
  seq?.play();
};
const onPause = () => {
  seq?.pause();
};
const onStop = () => {
  seq?.stop();
};
const onReplay = () => {
  seq?.play(true);
};
const onPlayAt20Seconds = () => {
  if (seq) {
    seq.currentTime = 110;
    seq.play();
  }
};

const onFileChange = async event => {
  // unpause context
  await context.resume();
  const file = event.target?.files?.[0];
  if (!file || !synth) return;
  const midiFile = await file.arrayBuffer();
  // create sequencer if it does not exist
  const song = [{ binary: midiFile, altName: "" }];
  if(seq === undefined){
    seq = new Sequencer(song, synth);
  } else {
    // else load a new song, avoid creating a new sequencer
    seq.loadNewSongList(song);
  }
  seq.loop = false;
  seq.addOnSongEndedEvent(() => {
    console.log("play ended");
  }, "_");
};

// LOADING NEW SOUNDS

// here's a function that dynamically loads a new timbre
const loadNewTimbre = async (timbreName: string) => {
    // check if the timbre is loaded in cache
    let desfont: ArrayBuffer = cachedTimbres[timbreName];
    if(desfont === undefined){
      // it is not. load it and cache it (if it's ever needed again)
      desfont = await (await fetch(`/soundfonts/${timbreName}.sf3`)).arrayBuffer();
      cachedTimbres[timbreName] = desfont;
    }

    // add (merge) the new soundfont into the synth
    await synth.soundfontManager.addNewSoundFont(desfont, timbreName);
    console.log("Loaded a new timbre!", timbreName);
}

// for example user selects a new soundfont, it should get added to the synth on top of the previous ones
const onSelectionChange = async event => {
    await loadNewTimbre(event.target.value);
}