search

Birch-san / juicysfplugin

Audio plugin (e.g. VST, AU) to play soundfonts on macOS, Windows, Linux
GNU General Public License v3.0
209 stars 27 forks source link

Too-large buffer sizes causes timing issues in REAPER #39

Open DaforLynx opened 2 years ago

DaforLynx commented 2 years ago

Using newest version, but not older versions, of JuicySF, if I change the block size to 1024 samples, I get weird timing issues when fast notes play. When I change it to 512 samples, the issues disappear (but I'm left with a 512 sample buffer, of course). I believe it happens with both VST and VST3 versions. I will need to do more testing (and recording) but it has caused me a few problems so far.

Birch-san commented 2 years ago

hey, that's interesting and worrying.

are you using the Windows release?

DaforLynx commented 2 years ago

Yes. If you want to check for yourself REAPER comes with a 60 day free trial

DaforLynx commented 2 years ago

Here's a demonstration. The first playthrough is 512 samples, the second is 1024, the third is both of them overlaid, the fourth is with phase inversion.

https://user-images.githubusercontent.com/13303368/165852609-587b286e-5653-49e7-8a5f-bf5b3d711158.mp4

I realize this isn't really a clear demonstration but there is definitely a discrepancy. With more complex rhythms it gets even more noticeable.

Birch-san commented 2 years ago

thanks for the demo. yeah, looks like a problem.

the main things that have changed since the last-known good release have been:

a very good option is to stay on the https://github.com/Birch-san/juicysfplugin/releases/tag/2.3.3 release. the only really impactful thing is the newer fluidsynth (which has bug fixes for things like sustain pedals).

there are even some advantages to 2.3.3:

if the problem can be reproduced on macOS, then that makes things a lot simpler. can you send the MIDI and the soundfont which reproduces the problem?

DaforLynx commented 2 years ago

Sure. The soundfont https://drive.google.com/file/d/1uDFibloXawmeRFR5maTRXNi2tbjEYKab/view?usp=sharing The midi file https://drive.google.com/file/d/1uoqKRuIH9tCdUarIu6gCyjUsH8DViriv/view?usp=sharing And even the REAPER file https://drive.google.com/file/d/1RP_W31PnOnCnQtqVOzsGdV7BgfwOGFiW/view?usp=sharing

DaforLynx commented 2 years ago

Another strange thing I noticed - program changes (and potentially other MIDI CC-related functions) work in 3.x, but not, for some reason, in 2.x. Maybe it's the FluidSynth upgrade. Also, I believe I'm running 3.0.0 instead of the latest version, so I will update and see if that fixes any issues.

DaforLynx commented 2 years ago

Updating did not fix this buffer issue.

DaforLynx commented 2 years ago

I've tested in Ableton Live 10, and the issue doesn't seem to be present. Ableton's a lot heavier than REAPER in general - sample buffer sizes that work fine in REAPER artifact in Ableton. The mystery now lies in how REAPER interacts with JuicySF 3.x to produce timing errors.

Birch-san commented 2 years ago

I think the audio difference is a bit too subtle for me to recognise whether the problem is reproducing.

how about sending a single note through a sine wave soundfont, render it (with 512 then 1024 buffer sizes), and see what the waveform looks like visually? if there are discontinuities or other telltale problems, that could give us a clue (especially if it there's anything interesting we can tell from reading off the time/duration of the problem).

https://github.com/TeensyAudio/Wavetable-Synthesis/blob/master/Soundfonts/Sine%20Wave.sf2

DaforLynx commented 2 years ago

Here's a folder containing some results I tested at 3 buffer sizes, once using repeating eighth notes and once with a single sustained note. The soundfont you linked wasn't quite a true sine wave, but I had something closer. https://drive.google.com/file/d/11heRAb2i4yOsltsblQQTrW3_zoJHUwr9/view?usp=sharing

DaforLynx commented 2 years ago

The sustained notes don't look very different, except that on the higher buffer sizes it ends earlier than with 512 samples... On the repeated notes you can clearly see differences where the wave ends one note and restarts on a new note. Again, the higher sample sizes are ending the note earlier than they should be, but they're also starting the next note earlier too.

Birch-san commented 2 years ago

thanks for rendering and sending those.
the sustain one is the simplest to reason about, so I'm starting with that one.

image

I've selected "one block's worth of audio" (1/44100*512 or 1/44100*1024 secs).

there's no obvious discontinuities at the end of the block, nor are there any patterns that seem to repeat at either 0.01160998 or 0.02321995 sec periods (for 512, 1024 respectively).

for reference, the kinds of things I'm looking for are like this error I got years ago. but that was a huge enough error as to be easily audible.

1024 sustain seems to end prematurely:

image

but it doesn't seem to be a case of "the whole waveform got accelerated", because all the waves seem to be perfectly in-phase the whole time (nodes and antinodes present in the same places).

it kinda looks like the decay envelope for the 1024 wave got triggered early. like 0.012secs early:

image

the "trigger" for the sample to decay is a MIDI note off.

0.012 secs happens to be the duration of "a block of 512 samples, sampled at 44100Hz". might not be a coincidence.

I wonder if it's something like "the timing for MIDI note off is getting rounded to 'the start of the processed block'"?
and, the bigger your blocks are (i.e. bigger buffers), the worse the rounding error can be?

I'm not super convinced by the theory (after all: the MIDI messages are associated with a time).

I wonder whether it's always a discrepancy of 0.012secs (or some multiple thereof).
and I wonder whether a longer note produces a wider discrepancy (I wouldn't expect it to though).

I also wonder whether there could be anything REAPER-specific (for example which MIDI messages it gives us and when, and what timing is associated with them).

I believe it happens with both VST and VST3 versions.

are we super sure that both versions have been tried? I configured it to present them as one single plugin if both are found. I think in that situation it prefers to show you VST3. one way to be certain is to move the unwanted variant out of your VST/VST3 folder and scan for plugins again.

I could try building a release which uses old fluidsynth and JUCE, and undoes the one-line change I made to processBlock(). if that fixes the problem, then that'd mean there's a way to fix this without throwing away the new build system. then we could narrow down which of the changes is the culprit. conversely, if it doesn't fix the problem: that'd suggest the problem lies with the new build system. which would be no bueno.

DaforLynx commented 2 years ago

image https://drive.google.com/file/d/1Ulu1Z2KN7Hk6JT_1kz-dNg5Ff-lwraJr/view?usp=sharing You may find it interesting what happens when I separate out the MIDI notes. It is doing a 32nd note every 16th note at 120 bpm. From the top, 2048, 1024, 512, 256 samples. Every so often (regularly?) on the beat the notes "catch up" to each other.

DaforLynx commented 2 years ago

I've also just confirmed it happens with different audio systems (DirectSound, WASAPI [which I've been using so far], ASIO). The note begins slightly before it should. Additionally, it doesn't seem to matter how often the notes are activated. image

DaforLynx commented 2 years ago

Oh, no. Oh, nonono. I've suddenly realized this isn't actually exclusive to JuicySF 3.x. It seems to affect older versions as well. Which means it could be a REAPER issue. I don't know why I came under the impression that it was a version-specific issue.

So, I've tested both VST2 and VST3 versions of 3.1.0, and they show the exact same behavior (they phase cancel) And when I replace the VST3 version with the version present in the folder before, which is older (not sure how much older, but it pops up a purple JUCE logo), it again phase cancels.

Sorry for the confusion, I'll bring it up with the REAPER folks instead. (although you're free to help figure it out if you want - because the case remains that so far JuicySF is the only sf2 plugin that causes these problems in REAPER) The only reason I hadn't noticed until recently was because I was never working on projects where this behavior was noticeable (i.e. there were only one or two instances of JuicySF and they were playing parts in which the behavior wasn't noticeable at the buffer size I used)

Birch-san commented 2 years ago

phew! thanks for clarifying.

I don't know how other SF2 plugins seem to be immune. JuicySF is pretty simple (just passes the MIDI messages to fluidsynth [0], which is pretty battle-tested).

yeah, I think worth taking up with the REAPER folks. what Juicy does is pretty simple, and nobody's reported this kind of issue in any other DAW. we've had DAW-specific issues before, so it's possible.

from the above analysis it kinda looks like we're receiving the wrong timestamps on NoteOff MIDI events? like the timestamp gets rounded to "start of the block".

[0]
https://github.com/Birch-san/juicysfplugin/blob/master/Source/PluginProcessor.cpp#L189
https://github.com/Birch-san/juicysfplugin/blob/master/Source/FluidSynthModel.cpp#L346-L350

DaforLynx commented 2 years ago

I don't think that explains everything, because not only do notes end early, they also start early, and seemingly randomly (if I export the same phrase twice they don't phase cancel - but two instances of JuicySF will).

I highly doubt it's a problem with JuicySF itself - it's probably an interaction between REAPER and Fluidsynth. I haven't seen any other SF2 plugins that use Fluidsynth, so I can't know for sure.

BHVXT commented 1 year ago

Just want to add that I'm having a similar issue with the x64 version of Ableton Live Intro 8.4.2. I have no idea if newer Ableton versions have the same issue (likely not), but it is absolutely a buffer timing issue - with very high buffer sizes (seemingly the default, around 4000 something samples) the timing bugs out like crazy, but when at a rate of, say, 1024 samples (21 ms), there's no real issue.

I've used the latest version of juicysfplugin and the older one linked in this thread and the results are pretty much the same (one is more buggy than the other - I'll have to look more into that).

Birch-san commented 1 year ago

thanks @BHVXT for the detail.

does the same problem happen on the older Windows release?
https://github.com/Birch-san/juicysfplugin/releases/tag/2.3.3

that one was built with older JUCE, older fluidsynth, and a different toolchain (MSVC) & build system (vsbuild).

mmolch commented 1 year ago

I just ran into this issue and here's what I found in reaper:

The DAW sends all MIDI (and automation) events that will occur within the timeframe of the buffer at the same time along with the buffer. E.g. if you have a one second buffer (e.g. buffer size of 44100 at a sample rate of 44100), the plugin receives a bundle of MIDI data every second when it receives the buffer.

The problem with this plugin is that it will play all events as soon as it receives the data along with the buffer. So with the above example you will get one combined sound every second. I'm neither familiar with this plugin nor fluidsynth, but the time stamps are obviously ignored.

While not a complete show-stopper, it is important to know that the buffer size here directly influences the MIDI resolution. I now render with a buffer size of 32, so the maximal error is 32/44100 seconds (below 1 ms)

Edit: If I see this correctly the time is actually completely ignored in FluidSynthModel::processBlock. It iterates through the events and triggers noteon, noteoff etc. events in fluidsynth but never actually takes the time into account.

Edit 2: The correct way would probably be to additionally run fluid_synth_process(...) in the MIDI event loop before triggering the event. Possibly related read: https://github.com/FluidSynth/fluidsynth/issues/1134