search

muse-sequencer / muse

MusE is a digital audio workstation with support for both Audio and MIDI
https://muse-sequencer.github.io/
Other
644 stars 69 forks source link

State handling in LV2/VST plugins #1068

Open theGreatWhiteShark opened 2 years ago

theGreatWhiteShark commented 2 years ago

When using some LSP plugins I experience odd behavior in case the plugin in bypassed.

To Reproduce Steps to reproduce the behavior:

  1. Load LSP Impulse Response as plugin and bypass it (happens with both LV2 and VST version)
  2. Open the GUI and load a sample file
  3. Alternatively open the GUI when bypassed and a sample file was already loaded

Expected behavior The waveform of the sample should be rendered in the plugin.

Desktop (please complete the following information):

Additional context At first I fought this is a bug in the plugin itself and filed a bug report there. But @sadko4u looked into it and it seems to be a problem with the internal state handling in Muse itself (analysis attached below).

OK, now I found out what's happening.

  1. The plugin DSP is instantiated by MuSE.
  2. Then, the state restoration happens and the state of all parameters becomes restored inside of the plugin (the plugin knows each setting and which file is associated with which sample placeholder).
  3. Because the host doesn't issue the run() method for LV2, the plugin has no possibility to detect that the state has been changed and trigger the process of loading of audio samples.
  4. Because no audio sample loading has triggered, you observe empty samples in the UI.
  5. When you enable plugin in a mixer strip, the plugin immediately loads all samples and displays them.

Interesting facts:

  1. LSP Plugins use hot bypass which is specified by this piece of TTL code: 
        [
            a lv2:InputPort, lv2:ControlPort ;
            lv2:index 4 ;
            lv2:symbol "enabled" ;
            lv2:name "Enabled" ;
            lv2:designation lv2:enabled ;
            lv2:portProperty lv2:toggled ;
            lv2:minimum 0 ;
            lv2:maximum 1 ;
            lv2:default 1 ;
    ]

    That means that plugin controls bypass process by itself and host should not prevent the plugin from executing the run() method. Anyway, here's more detailed description of enabled port designation: https://lv2plug.in/ns/lv2core#enabled

  2. Muse doesn't allow to properly restore the special KVT (Key-Value Tree) object introduced by LSP from the plugin state. Seems that it doesn't store URIs when serializing state and doesn't map them when deserializing state. That crashed most recent version of LSP Multisampler and I was required to add a workaround for the case when the host can not unmap the URID. Here's and example of such KVT object from Ardour's plugin state description: 
    
@prefix atom: <http://lv2plug.in/ns/ext/atom#> .
@prefix lv2: <http://lv2plug.in/ns/lv2core#> .
@prefix pset: <http://lv2plug.in/ns/ext/presets#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix state: <http://lv2plug.in/ns/ext/state#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<> a pset:Preset ; lv2:appliesTo http://lsp-plug.in/plugins/lv2/multisampler_x12 ; state:state [ http://lsp-plug.in/plugins/lv2/multisampler_x12/KVT [ a http://lsp-plug.in/types/lv2/types#KVT ; http://lsp-plug.in/ui/lv2//instrument/9/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "" ] ; http://lsp-plug.in/ui/lv2//instrument/8/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "" ] ; http://lsp-plug.in/ui/lv2//instrument/7/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "123" ] ; http://lsp-plug.in/ui/lv2//instrument/6/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "" ] ; http://lsp-plug.in/ui/lv2//instrument/5/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "" ] ; http://lsp-plug.in/ui/lv2//instrument/4/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "asdfsdf" ] ; http://lsp-plug.in/ui/lv2//instrument/3/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "" ] ; http://lsp-plug.in/ui/lv2//instrument/2/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "sdfsdf" ] ; http://lsp-plug.in/ui/lv2//instrument/11/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "sdfdf" ] ; http://lsp-plug.in/ui/lv2//instrument/10/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "" ] ; http://lsp-plug.in/ui/lv2//instrument/1/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "" ] ; http://lsp-plug.in/ui/lv2//instrument/0/name [ a http://lsp-plug.in/types/lv2/types#KVTProperty ; http://lsp-plug.in/types/lv2/KVTProperty#flags "0"^^xsd:int ; http://lsp-plug.in/types/lv2/KVTProperty#value "sdfsdf" ] ] ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#inst "7"^^xsd:int ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#chan_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#note_0 "9.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#oct_0 "4.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#mgrp_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#mtg_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#nto_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#trg_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#dyna_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#drft_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#ssel_0 "0"^^xsd:int ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#hc_0_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#tc_0_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#fi_0_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#fo_0_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#mk_0_0 "1.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#vl_0_0 "100.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#pd_0_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#on_0_0 "1.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#ls_0_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#rs_0_0 "0.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#pl_0_0 "-100.0"^^xsd:float ; http://lsp-plug.in/plugins/lv2/multisampler_x12/ports#pr_0_0 "100.0"^^xsd:float ] .

3. I've built MuSE from source and it dies when launching and closing the application without any changes at this moment:

0 0x00007fffed736360 in raise () at /lib64/libc.so.6

1 0x00007fffed737941 in abort () at /lib64/libc.so.6

2 0x00007fffed7797b7 in __libc_message () at /lib64/libc.so.6

3 0x00007fffed77ffd3 in malloc_printerr () at /lib64/libc.so.6

4 0x00007fffed7804b2 in malloc_consolidate () at /lib64/libc.so.6

5 0x00007fffed782070 in _int_free () at /lib64/libc.so.6

6 0x000055555577b832 in MusECore::LV2Synth::~LV2Synth() ()

7 0x000055555577ba89 in MusECore::LV2Synth::~LV2Synth() ()

8 0x000055555570a115 in MusECore::Song::cleanupForQuit() ()

9 0x0000555555659fd0 in MusEGui::MusE::closeEvent(QCloseEvent*) ()

10 0x00007ffff3d91238 in QWidget::event(QEvent*) () at /usr/lib64/libQt5Widgets.so.5

11 0x00007ffff3e9ad70 in QMainWindow::event(QEvent*) () at /usr/lib64/libQt5Widgets.so.5

12 0x00007ffff3d50f2c in QApplicationPrivate::notify_helper(QObject, QEvent) () at /usr/lib64/libQt5Widgets.so.5

13 0x00007ffff3d58540 in QApplication::notify(QObject, QEvent) () at /usr/lib64/libQt5Widgets.so.5

14 0x0000555555650bd8 in MuseApplication::notify(QObject, QEvent) ()

15 0x00007ffff2ecb328 in QCoreApplication::notifyInternal2(QObject, QEvent) () at /usr/lib64/libQt5Core.so.5

16 0x00007ffff3d8c745 in QWidgetPrivate::close_helper(QWidgetPrivate::CloseMode) () at /usr/lib64/libQt5Widgets.so.5

17 0x00007ffff3dab5e6 in () at /usr/lib64/libQt5Widgets.so.5

18 0x00007ffff3daee14 in () at /usr/lib64/libQt5Widgets.so.5

19 0x00007ffff3d50f2c in QApplicationPrivate::notify_helper(QObject, QEvent) () at /usr/lib64/libQt5Widgets.so.5

20 0x00007ffff3d58540 in QApplication::notify(QObject, QEvent) () at /usr/lib64/libQt5Widgets.so.5

21 0x0000555555650bd8 in MuseApplication::notify(QObject, QEvent) ()

22 0x00007ffff2ecb328 in QCoreApplication::notifyInternal2(QObject, QEvent) () at /usr/lib64/libQt5Core.so.5

23 0x00007ffff34cf7f5 in QGuiApplicationPrivate::processCloseEvent(QWindowSystemInterfacePrivate::CloseEvent*) () at /usr/lib64/libQt5Gui.so.5

24 0x00007ffff34d37e5 in QGuiApplicationPrivate::processWindowSystemEvent(QWindowSystemInterfacePrivate::WindowSystemEvent*) () at /usr/lib64/libQt5Gui.so.5

25 0x00007ffff34ad23b in QWindowSystemInterface::sendWindowSystemEvents(QFlags) () at /usr/lib64/libQt5Gui.so.5

26 0x00007fffe02e21da in () at /usr/lib64/libQt5XcbQpa.so.5

27 0x00007ffff1f05624 in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0

28 0x00007ffff1f059c0 in () at /usr/lib64/libglib-2.0.so.0

29 0x00007ffff1f05a4c in g_main_context_iteration () at /usr/lib64/libglib-2.0.so.0

30 0x00007ffff2f283af in QEventDispatcherGlib::processEvents(QFlags) () at /usr/lib64/libQt5Core.so.5

31 0x00007ffff2ec957a in QEventLoop::exec(QFlags) () at /usr/lib64/libQt5Core.so.5

32 0x00007ffff2ed2780 in QCoreApplication::exec() () at /usr/lib64/libQt5Core.so.5

33 0x0000555555643187 in main ()



So, there are several problems related to the plugin lifecycle but I don't think that LSP is the reason by itself. I think MuSE developers should look at the problems I've mentioned here.

_Originally posted by @sadko4u in https://github.com/sadko4u/lsp-plugins/issues/205#issuecomment-1105772331_
theGreatWhiteShark commented 2 years ago

FYI @sadko4u updated the KVT description to better handle these type of problems

@prefix atom: <http://lv2plug.in/ns/ext/atom#> .
@prefix lv2: <http://lv2plug.in/ns/lv2core#> .
@prefix pset: <http://lv2plug.in/ns/ext/presets#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix state: <http://lv2plug.in/ns/ext/state#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<>
    a pset:Preset ;
    lv2:appliesTo <http://lsp-plug.in/plugins/lv2/multisampler_x12> ;
    state:state [
        <http://lsp-plug.in/plugins/lv2/multisampler_x12/KVT> [
            a atom:Tuple ;
            rdf:value (
                [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/0/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "1-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/1/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "2-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/10/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "11-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/11/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "12-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/2/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "3-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/3/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "4-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/4/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "5-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/5/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "6-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/6/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "7-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/7/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "8-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/8/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "9-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ] [
                    a <http://lsp-plug.in/types/lv2/KVTEntry> ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#key> "/instrument/9/name" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#value> "10-1" ;
                    <http://lsp-plug.in/types/lv2/KVTEntry#flags> "0"^^xsd:int
                ]
            )
        ]
    ] .
terminator356 commented 1 year ago

Hi. Sorry for the very late reply. Hm, that's a tough one. Yep, when we say 'bypass' we really mean it - by not running it at all. That was part of the concept, to avoid unnecessary or unknown excess CPU time inside some plugin's run, while we bypass the audio around it. Maybe we'll have to allow a run at least once whenever something major changes.

The enable port, hm that's a tough one as well, because obviously since not every plugin has such a port, we can't rely on that. What would help is if LV2 had such an 'enable' port type designator so that we could reliably ask LV2 if a plugin has this port. (Similar to latency ports etc.)

terminator356 commented 1 year ago

Those items and KVT will require more investigation. For me I must prioritize this low at the moment due to other branches and bug fixes. I will try to revisit... Keeping it open for now...

theGreatWhiteShark commented 1 year ago

For me I must prioritize this low at the moment due to other branches and bug fixes. I will try to revisit... Keeping it open for now...

Sounds legit. Thanks for the update.

terminator356 commented 1 year ago

What would help is if LV2 had such an 'enable' port type designator

lv2:designation lv2:enabled

Oops, sorry I missed that, and thanks for the link! This is good. Coincidentally I'm working in this area right now dealing with a semi-related issue. I can't promise anything immediately but if I do this right we can have a fix.

One question I had, is whether it is acceptable in this situation to simply execute a run with zero length (which is permitted by the specs, although they only mention that it is for updating output ports such as latency, not for this scenario).

theGreatWhiteShark commented 1 year ago

One question I had, is whether it is acceptable in this situation to simply execute a run with zero length (which is permitted by the specs, although they only mention that it is for updating output ports such as latency, not for this scenario).

Unfortunately I'm just the messenger and can't help with any LV2 related matter.