Missing MIDI notes in audio [macOS VST instrument]

[ThomasLee-git]

Thanks for creating such an useful tool~ I've had these problems the other day when trying to render a midi file using free spitfire audio labs.

1. I have to do time.sleep(0.5) before calling engine.render() to get "valid audio", otherwise an empty audio is returned.
2. Even if step 1 is done, only part of the "expected" audio is returned. Many midi notes are not rendered properly.

My question is that is there a way for dawdreamer to make sure vst plugin samples are ready for rendering and the returned audio is the expected result of the midi?

Any help is appreciated~

I'm using macos 11.4 intel, pip installed dawdreamer v0.5.8.2, and test code looks like

import dawdreamer as daw
import time
from pathlib import Path

SAMPLE_RATE, BUFFER_SIZE = 44100, 512
sleep_time = 0.5
midi_path = Path("path_to_midi.mid")
SYNTH_PLUGIN = Path("/Library/Audio/Plug-Ins/VST/LABS.vst")
preset_path = Path("path_to_piano.fxp")
output_path = Path("path_to_wav")
engine = daw.RenderEngine(SAMPLE_RATE, BUFFER_SIZE)
synth = engine.make_plugin_processor("my_synth", SYNTH_PLUGIN.as_posix())
assert synth.load_preset(preset_path.as_posix())
time.sleep(sleep_time)
assert synth.load_midi(midi_path.as_posix())
print("num_midi_event", synth.n_midi_events)
graph = [
    (synth, []),
]
engine.load_graph(graph)
engine.render(20.0)  # Render 10 seconds audio.
audio = engine.get_audio()  # Returns numpy.ndarray shaped (2, NUM_SAMPLES)
save_wav(output_path.as_posix(), audio)

[DBraun]

Hi thanks. I'll look into this plugin in the near future. What you describe hasn't been an issue in the other plugins I've tested, but I can see it being possible because I think people reported it with RenderMan too.

The answer to your question is that the closest thing is the ability to inspect the number of midi notes that have been loaded. synth.n_midi_events == 10*2 etc for 10 MIDI notes. Each note has two events ("on" and "off"). Note that when you load directly from MIDI, I think all kinds of MIDI messages can be sent to the plugin. In this line of code, https://github.com/DBraun/DawDreamer/blob/772048dcbfb2ceb2519d9c6a77917e25970ca1db/Source/PluginProcessor.cpp#L151 it's not checking what the type of MIDI message is (on/off). It could be related to CC or other. I only just realized this the other day and will document it better or improve it somehow.

[ThomasLee-git]

Thanks for such an instant reply~

I did some homework myself to check if any midi event is missed during load_midi() and render(). The midi is exported from a DAW and DOES contain many non-on/off events. I'll give it a try whether the engine works properly after filtering out those note-unrelated events.

Waiting for your updates~

[DBraun]

I still need to check out that plugin, but version 0.6.0 has a newer load_midi method for Plugin Processor.

https://github.com/DBraun/DawDreamer/blob/69e885a94f13cf5a4110571b9c69086707e90a5b/tests/test_plugins.py#L139

[kyungyunlee]

Experienced same problem here when trying to render with LABS/Spitfire vst. Time sleeping seems to help for now...

[DBraun]

Good news! I tried out Spitfire LABS. It's working fine with the new open_editor() function. Just enable this line and try this function: https://github.com/DBraun/DawDreamer/blob/9d286e1171b7f4c89558e5bdb9337649d2803f53/tests/test_plugins.py#L379

Also change the plugin path. I'm using plugin_path = "C:/VSTPlugIns/LABS (64 Bit).dll" on Windows

I can use the editor, save the state, and then load it later without extra calls to open_editor. Let me know how it goes for you.

[kyungyunlee]

@DBraun Hi thanks for checking it out !! But unfortunately it's still not working. The plugin itself works fine in Reaper, but when I use the open_editor() and click the keyboard from the UI, I am not hearing any sounds. Also, is the "states" same as "presets"?

p.s I am on intel macOS

[DBraun]

Maybe it's a macOS issue then. I'll try soon. Keyboard input isn't working yet. Did you select a preset and then click load? That was enough for me.

States are kind of like presets, but I'm referring to the new methods:

if isfile(state_file_path):
    synth.load_state(state_file_path)
synth.open_editor()
synth.save_state(state_file_path)

If you make a change with open_editor and save it, then you can call load_state next time and not call open_editor.

[kyungyunlee]

@DBraun Hi thanks for the fast reply! I've found out that LABS VST3 doesn't work but VST works fine. For other VSTs, VST3 version seems to work though. Same situation with the load_preset function.

[DBraun]

Ok, I will close here and migrate the discussion to https://github.com/DBraun/DawDreamer/issues/86

[ThomasLee-git]

@DBraun Many thanks~ Based on my understanding, the open_editor function solves the silence problem by opening UI window, which gives VST the time for loading samples and other startup things. But, sorry for the but, I have to close UI manually to continue the rendering. This breaks the capability of automatically rendering thousands of midi files in the background.

Is it possible to do equivalent operation in a headless environment?

[DBraun]

The idea is to only use open_editor once, save the state, and then call load_state without extra calls to open_editor. The file extension doesn't matter because it's just binary. Unfortunately you will have to do the work of creating these "state" files once per state.

import os

def load_help(processor, filepath: str):

    """
    Load a state if it exists.
    Otherwise open the UI to have the user select/create it. Then it will be saved.
    """

    if os.path.isfile(filepath):
        processor.load_state(filepath)
    else:
        # This will interrupt a headless process
        processor.open_editor()
        # Save it for next time
        processor.save_state(filepath)

load_help(synth, "path/to/awesome-preset-state")

[ThomasLee-git]

I'm pretty sure your idea is well delivered, but everything goes back to it was after saving state first and then loading it by running the script twice. Only empty audio is returned. Would @kyungyunlee please testify this result on Mac? Or maybe it is just the difference of VST implementation mechanism between Win and Mac.

[kyungyunlee]

@ThomasLee-git Sorry, I didn't quite get what you want me to check. Could you clarify? Btw in my case, I switched to using VST, instead of VST3, and managed to render. I am using Mac with intel chip.

[ThomasLee-git]

@kyungyunlee Sorry for not being clear~ Here are the steps to reproduce (I modified a little bit to suit my case, and the following code will work if you installed spitfire labs with default configurations).

1. run the script for the first time, then UI window will show up to let you select an instrument, soft piano for example, and labs_state will be saved after closing UI.
2. run the script again, this time "test.wav" will be saved and it's empty on my machine.

I have tried selecting different instruments, but the result is the same. If I keep running the script with labs_state present, after a cern time notes at the end of midi file are rendered , which goes back to the state of my original post. The time when notes start to occur is totally random.

import dawdreamer as daw
import numpy as np

from pathlib import Path
import wave

SAMPLE_RATE = 44100
BUFFER_SIZE = 512

def save_wav(path: str, audio: np.ndarray) -> None:
    assert len(audio.shape) == 2
    audio = (audio * (2**15 - 1)).astype(np.int16)
    with wave.open(path, mode="w") as wf:
        wf.setnchannels(audio.shape[0])
        wf.setsampwidth(2)
        wf.setframerate(SAMPLE_RATE)
        wf.writeframes(audio.T.tobytes())

if __name__ == "__main__":
    duration_to_render = 8.0
    midi_path = Path("test.mid")
    SYNTH_PLUGIN = Path("/Library/Audio/Plug-Ins/VST/LABS.vst")
    state_path = Path("labs_state")
    output_path = Path("test.wav")
    engine = daw.RenderEngine(SAMPLE_RATE, BUFFER_SIZE)
    synth = engine.make_plugin_processor("my_synth", SYNTH_PLUGIN.as_posix())
    if not state_path.exists():
        synth.open_editor()
        synth.save_state(state_path.as_posix())
    else:
        synth.load_state(state_path.as_posix())
        synth.load_midi(midi_path.as_posix(), all_events=False)
        print("num_midi_events", synth.n_midi_events)
        graph = [
            (synth, []),
        ]
        engine.load_graph(graph)
        engine.render(duration_to_render)
        audio = engine.get_audio()
        save_wav(output_path.as_posix(), audio)

test.mid.zip

[kyungyunlee]

@ThomasLee-git

Here's the order of tracks in the image

1. Test midi
2. Wav rendered with LABS.vst
3. Wav rendered with LABS.vst + time.sleep after loading state
4. Wav rendered with LABS.vst3

I tested with both VST and VST3. While VST3 renders more notes than VST, both of them don't fully render. Adding time.sleep() appears to solve the problem for VST and VST3 , but when I tried in the past, time sleep didn't always guarantee perfect rendering.

[ThomasLee-git]

@kyungyunlee Thank you so much for the verification~Really glad that we are finally on the same page!

I'm no VST expert and know little about VST implementation. Base on my knowledge, VST 2/3 standards have nothing related to synchronization and it's totally up to the vender whether a buffer can be rendered or dropped, right? I'm curious about what other DAWs do to ensure perfect rendering.

[rupertsworld]

I was having a problem with this exact same plugin – turns out my issue was related to the fact that this instrument generates 32-channel output, and I was trying to also use an instrument with 2-channel (stereo) output. When mixing these two outputs together, DawDreamer seemed to get upset and produce no output without throwing an error.

I'll add another issue for this when I get a chance. I managed to fix it by specifically getting two channels of audio from each instrument, then combining them by adding the arrays together in numpy (I created a dictionary of multiple engines to make this work):

audio = self.engines[name].get_audio().transpose()[:, [0, 1]]
audios.append(audio)

mixed_audio = audios.pop()
for audio in audios:
            mixed_audio = mixed_audio + audio

wavfile.write(f"outputs/mixed_audio.wav", config["sample_rate"], mixed_audio)

[DBraun]

@rupertparry Can you try synth.set_bus(0, 2)? Then everything should work without warnings.

[rupertsworld]

@rupertparry Can you try synth.set_bus(0, 2)? Then everything should work without warnings.

When I attempt that, I get the following message:

ValueError: piano CANNOT ApplyBusesLayout inputs: 0 outputs: 2

The plugin is "LABS" by Spitfire Audio (https://labs.spitfireaudio.com/)

[DBraun]

@rupertparry That's interesting because on both macOS (v 1.5.1) and windows for me, LABS seems to default to 2 outputs. This is how I use Faust as a last resort to turn an N channel output into 2:

https://github.com/DBraun/DawDreamer/blob/eb845cb9e97be8564b411a0e2c5c95ef458cc455/tests/test_plugins.py#L339-L344

However, this code isn't even executing for LABS on my mac or windows because num_outputs is already 2.

Please open another issue if you're ready.

[DBraun]

Sorry I still need to work on this. LABS is showing the same problem.