Ideas to handle clicking when looping and custom soundfonts

[this-fifo]

Hey @danigb, I wanted to first thank you for your incredible work! This project, as well as your other work, is truly amazing and inspiring!

Alright, now, I have a few things to discuss so I'll introduce some background context first:

I want to use my custom soundfonts in Web Audio related projects.

Initially, I came across jet2jet/js-synthesizer, which is a fantastic project and worked liked a charm until ... well ... until it didn't.

Essentially the issue I had with it is that it doesn't allow for a nice way to share the AudioContext with standardized-audio-context (Hard requirement for me, as I have several other audio nodes that rely on that)

I want to design and use my own soundfonts.

That was super easy with jet2jet/js-synthesizer since it allows me to load the raw .sf2 file in its web assembly module approach.

I noticed with smplr, however, you seem to have adopted a standard of pre-rendered samples, without the need for web assembly parsers that can read the binary .sf2 but rather use raw .mp3 / .ogg files through data:audio/mpeg;base64, encodings.

I think that this is an acceptable trade-off, I read through how this happens for smplr and ended up landing on this soundfont_builder.rb script.

Using that script was remarkably easy, and I managed to generate my custom soundfonts in a smplr friendly format, so that's cool!

Except, it looks like smplr took an approach of getting the loop points from this process @goldst came up with through this generate-loop-data.js script. — This is where I was a bit lost to be honest, I think it would be nice to integrate that with the soundfont_builder.rb approach from @gleitz but I'm not sure how to do it currently, would appreciate some advice here 😅

PS: Do correct me if any of what I said is nonsense.

Looping!

I want looping, so I can have long sustained notes. Like an 8 second pad chord.

Despite not having a nice mycustomsoundfont-loop.json that smplr can automatically load for with loadLoopData: true, it looks like I can still achieve looping by doing:

const pads = new Soundfont(context, {
    instrumentUrl: './pads/acoustic_grand_piano-ogg.js', // ignore the acoustic_grand_piano name, that's because soundfont_builder.rb turns program 0 into that and I didn't rename it
    loadLoopData: true, // this param is effectively a noOp since I don't have a -loop.json file
})
["C4", "E4", "G4", "A4"].forEach((note, i) => {
  pads.start({
    note,
    duration: 8,
    loop: true,
    loopStart: 0.25, // fine tuned manually
    loopEnd: 2.85, // fined tuned manually
  });
});

Interestingly enough, my actual loop point data from Polyphony was 13926 and 176400 over 44100 samples which translates into 0.33 and 4 seconds I believe, but the samples created by soundfont_builder.rb seem to be capped at 3 seconds so it didn't even make sense to use them. — Would appreciate some advice on this as well 🙇‍♂️

Another thing I noticed is that the way you load the loop points seem to assume a sample rate of 41000 but a lot of the soundfonts are using 44100

https://github.com/danigb/smplr/blob/74520503f9d175f8ce94eced163f9256065ae1d8/src/soundfont/soundfont-loops.ts#L25

I would assume part of the reason you're getting clicks in some instruments is due the sample rate mismatch and potential downsampling from conversion loss during this step in soundfont_builder.rb

https://github.com/gleitz/MIDI.js/blob/1433d3913d26c1e5f80b3fa0ab63c98584b0087d/generator/ruby/soundfont_builder.rb#L405-L407

Alright, what next

That all said, I was super happy that that worked and I could hear my custom soundfont playing a chord for a whole 8 seconds!

Finally, what I want to know is related to sometimes getting clicking or the feeling of the note being interrupted. I noticed in the docs you say "This feature is still experimental and can produces clicks on lot of instruments."

Why do you think that this is happening? How can I help or what do you recommend looking into to enable better support for looping?

I am wondering if it could also be a clock/sync issue, or if we can soften this with some attack/decay parameters in an envelope filter.

Happy to connect and chat more about this if you want too!

[danigb]

Wow @this-fifo this is amazing. Very well presented all the points and problems 👏 Inspiring too 👍

First of all, the sample rate of 41000 is totally a bug, and could be the reason behind the clicks! I'll start by fixing that and test. Let me do it right now.

About the script: I didn't use it myself, but I'll give it a try. I don't have any idea why it generates samples <3s length. Do you have any sf file I can use for testing?

Keep you posted

[this-fifo]

Hey @danigb thank you so much for getting back to me so quickly, I'm glad that I was able to help you find that bug!

In terms of the soundfont_builder.rb script, the reason why it caps at 3 seconds is because that that is the default configuration in that file, there are 2 hardcoded constants of interest I found there

VELOCITY = 85
DURATION = Integer(3000)

https://github.com/gleitz/MIDI.js/blob/1433d3913d26c1e5f80b3fa0ab63c98584b0087d/generator/ruby/soundfont_builder.rb#L352-L353

That constant is used to create a MIDI event that lasts from 0 to 3 with velocity 85 this way:

track.events << NoteOn.new(0, note_value, VELOCITY, 0)
track.events << NoteOff.new(0, note_value, VELOCITY, DURATION)

https://github.com/gleitz/MIDI.js/blob/1433d3913d26c1e5f80b3fa0ab63c98584b0087d/generator/ruby/soundfont_builder.rb#L393-L394

Later, another thing that I'm not sure why it's done that way but, when it renders the audio with fluidsynth, it also sets the gain to 0.5 with -g 0.5 here: https://github.com/gleitz/MIDI.js/blob/1433d3913d26c1e5f80b3fa0ab63c98584b0087d/generator/ruby/soundfont_builder.rb#L405

As for test soundfonts, I'm using various I'm finding on this website https://musical-artifacts.com/artifacts?utf8=%E2%9C%93&formats=sf2 some are really cool like this Supersaw Collection

How did your tests go after the sample adjustment? While I think that that's a step forward, I think we should discuss ideas to incorporate loop points in the same script that renders the soundfonts into audio samples!

Appreciate any guidance or suggestions on what you think I should do to help here, I'm super excited about this project and definitely want to use it more!

[danigb]

In my initial tests the clicks seem to have disappeared, but I have not had time to test all the instruments.

Regarding your problem, I don't know how much automation you need. In case you don't need to convert many sf2 files it might be better to do it manually.

I have tested https://www.polyphone-soundfonts.com/download to extract the wave files. It works quite well. I've put the wav files into a repository: https://github.com/smpldsnds/supersaw

The next step would be to extract the sf2 information into json or another readable format and then to something smplr can read. I didn't have time to do it, but it seems there are several tools to convert sf2 files to json. I'm very open to PRs if you find one.

Good luck!

[danigb]

Hi @this-fifo

I've found this https://github.com/Mrtenz/soundfont2/ that looks very promising to convert sf2 files into smpr json.

[this-fifo]

Wow, thank you @danigb !

This is really promising, I was able to parse a raw soundfont in the browser and get a nice structure to work with!

I think we could totally use that to convert the soundfont into a format smplr likes, but I'm also curious if we can find a way to just natively incorporate the soundfont without the need to pre-render it into samples ranging from A0 to G7.

I noticed, for example, that package extracts the pure samples from the Soundfont!

Here's an example from that Supersaw.sf2, it shows the same data I can see in Polyphone!

The sample data is also available at an instrument level, so we could get creative here to support it all the way!

That said, I see two potential things of value:

1. Take that and figure out how to convert/map into existing json/js structure that smplr likes.
2. New separate implementation for accepting raw sf2 into smplr, without the need for them to be pre-rendered!

PS: I should say this is also my first time working with soundfonts and things like this so apologies in advance as I am learning as I go🙇‍♂️

[this-fifo]

I've just found how this is used in another related project, I think there's great inspiration and ideas to take from there:

• https://github.com/felixroos/sfumato/blob/main/src/index.ts
• https://loophole-letters.vercel.app/soundfonts (blog post from the author explaining how he did it)

[danigb]

Nice! Block post and Felix work looks great! I'd say the loading a sf2 directly into smplr would be the best option! I'll take a look when I have time

[danigb]

Hi @this-fifo I've created a new sampler capable of reading .sf2 files directly. Only limited amount of features are implemented. Feel free to open another issue if you need any of them.

[this-fifo]

Thank you so much @danigb , you're truly amazing! 🙇‍♂️

I've tested it with my custom soundfonts and I've been pretty happy so far!

Also, I took the liberty to fix a small typo you had on the README.md for it through this PR https://github.com/danigb/smplr/pull/82 — feel free to disregard it or correct it in your own words.

Thank you so much once again!