Crash while opening VST instrument plugin UI

travis-anderson commented 1 week ago

System Info:

NixOS: building from source using the nix package definition recently merged into nixpkgs-unstable
Wayland window backend (hyprland with XWayland enabled)

Problem: ossia-score crashes entirely when I click to open the plugin UI of any VST plugin that's been added to the timeline.

Running from a terminal, I get the following message printed at the moment of crash.

X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  1 (X_CreateWindow)
  Resource id in failed request:  0x4
  Serial number of failed request:  54
  Current serial number in output stream:  56

Being NixOS, I'm aware that this may be a misconfiguration on my part, but would appreciate advice, if so.

jcelerier commented 1 week ago

Hm, and I imagine this works in other VST hosts ? Which plugins did you try ? XCreateWindow failing may be would be caused by hyprland refusing to create it, I can try to install it on my end and see why it fails

jcelerier commented 1 week ago

hm, tried Hyprland and on ArchLinux it seems to work (even if it applies an absolutely awful UI scaling)

jcelerier commented 1 week ago

Would you be able to give a stack trace of where the crash happen ? maybe I can at least make it not crash in this case...

travis-anderson commented 1 week ago

Yes, other hosts work as expected with VSTs.

This is the full console output while running ossia up until the crash. There's no stack trace exactly, but the 3 distinctly different sections of logging are from starting up, starting a new project, then adding Dexed VST to the timeline, then trying to open its UI.

QVariant::load: unknown user type with name TimeVal.
pd 0.55.0
bonk version 1.5
fiddle version 1.1 TEST4
pd~ version 0.54
pique 0.1 for PD version 23
qt.qml.typeregistration: Invalid QML element name "Type"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Access"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Bounding"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Repetitions"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Duration"; value type names should begin with a lowercase letter
Debug: "https://raw.githubusercontent.com/ossia/score-addons/master/remote.json"   "404: Not Found" (:0)
Debug: Could not save device (:0)
Debug: Audio device emits changed: Dataflow::AudioDevice(0x55d473f59870) (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.1", "comment")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.1", "label")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.1", "trigger")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.1", "comment")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.1", "label")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.1", "trigger")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.1", "comment")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.1", "label")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.1", "expression")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.2", "comment")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.2", "label")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.2", "trigger")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "comment")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "label")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "min")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "max")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "default")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "percentage")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "speed")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "muted")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.2", "comment")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.2", "label")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.2", "trigger")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.2", "comment")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.2", "label")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.2", "expression")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.3", "comment")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.3", "label")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.3", "trigger")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "comment")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "label")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "min")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "max")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "default")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "percentage")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "speed")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "muted")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.wand39", "processes", "Dexed", "comment")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.wand39", "processes", "Dexed", "label")  -- last found:  intervals 
 (:0)
X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  1 (X_CreateWindow)
  Resource id in failed request:  0x6
  Serial number of failed request:  54
  Current serial number in output stream:  56

I'll do some trial-and-error with different configurations today and report here, too.

jcelerier commented 1 week ago

could you run ossia-score under gdb, and when the crash happens type "bt" and paste me what it says ? it will help pinpoint the error. from a terminal:

$ gdb ossia-score
... gdb starts ...
(gdb) r
... app launches ...
... crash happens ...
(gdb) bt

travis-anderson commented 1 week ago

...
[New Thread 0x7fff8d3fe6c0 (LWP 16941)]
[New Thread 0x7fff8cbfd6c0 (LWP 16942)]
[Thread 0x7fff8d3fe6c0 (LWP 16941) exited]
[New Thread 0x7fff8d3fe6c0 (LWP 16943)]
X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  1 (X_CreateWindow)
  Resource id in failed request:  0x5
  Serial number of failed request:  54
  Current serial number in output stream:  56
[Thread 0x7fffc9ffb6c0 (LWP 16902) exited]
[Thread 0x7fffe11be6c0 (LWP 16876) exited]
[Thread 0x7fffe91ff6c0 (LWP 16869) exited]
[Thread 0x7fff8cbfd6c0 (LWP 16942) exited]
[Thread 0x7fff8dbff6c0 (LWP 16939) exited]
[Thread 0x7fff8effd6c0 (LWP 16932) exited]
[Thread 0x7fffa4db66c0 (LWP 16931) exited]
[Thread 0x7fff8ffff6c0 (LWP 16930) exited]
[Thread 0x7fff8f7fe6c0 (LWP 16929) exited]
[Thread 0x7fffa5db86c0 (LWP 16928) exited]
[Thread 0x7fffa65b96c0 (LWP 16919) exited]
[Thread 0x7fffa6dba6c0 (LWP 16918) exited]
[Thread 0x7fffa75bb6c0 (LWP 16917) exited]
[Thread 0x7fffc8ff96c0 (LWP 16907) exited]
[Thread 0x7fffc97fa6c0 (LWP 16906) exited]
[Thread 0x7fffca7fc6c0 (LWP 16901) exited]
[Thread 0x7fffcaffd6c0 (LWP 16900) exited]
[Thread 0x7fffcb7fe6c0 (LWP 16899) exited]
[Thread 0x7fffe09586c0 (LWP 16898) exited]
[Thread 0x7fffcbfff6c0 (LWP 16897) exited]
[Thread 0x7fffe19bf6c0 (LWP 16875) exited]
[Thread 0x7fffe21c06c0 (LWP 16874) exited]
[Thread 0x7fffe29c16c0 (LWP 16873) exited]
[Thread 0x7fffe31c26c0 (LWP 16872) exited]
[Thread 0x7fffe3fff6c0 (LWP 16871) exited]
[Thread 0x7fffe89726c0 (LWP 16870) exited]
[Thread 0x7fffe96c52c0 (LWP 16852) exited]
[Thread 0x7fff8d3fe6c0 (LWP 16943) exited]
[New process 16852]
[Inferior 1 (process 16852) exited with code 01]
(gdb) bt
No stack.

jcelerier commented 1 week ago

hmm, I see that this call is happening inside the pug-ins - on my side I just do the following:

ERect* vstRect{};
fx.dispatcher(&fx, effEditGetRect, 0, 0, &vstRect, 0.f);
fx.dispatcher(&fx, effEditOpen, 0, 0, (void*)winId(), 0); // the crash happens here which is a call to within the VST

travis-anderson commented 1 week ago

Yeah, it is curious that only VSTs lead to this issue. I tried running without xwayland to see what other issues might occur.

In this case (no xwayland), the program crashes when I try to add VSTs to the timeline. Other processes attach just fine, but when adding a VST the app closes with the message: Failed to connect to the X Server. (Not unsurprising, but special)

travis-anderson commented 1 week ago

Further explorations:

_running ossia-score with an invalid $WAYLAND_DISPLAY (WAYLAND_DISPLAY=invalid ossia-score)_

new message line printed in the console along with the usual messages: qt.qpa.plugin: Could not load the Qt platform plugin "wayland" in "" even though it was found.
Program is functional, but I cannot click-and-drag Processes onto the timeline. Other click-drag operations, like branching the timeline, work fine.
However, if I recover or load a project which already has a VST attached, then the UI will load without crash

_running ossia-score with QT_QPA_PLATFORM=xcb ossia-score_

acts the same as invalid $WAYLAND_DISPLAY as above, but without the qt.qpa.plugin message

running from AppImage (appimage-run ossia.score-3.2.4-linux-amd64.AppImage)

acts the same as invalid $WAYLAND_DISPLAY as above, but with the following additional messages

[2024-07-09 10:22:37.813] [ossia] [error] No NDI library found on the system.
Please install an NDI library, for instance https://code.videolan.org/jbk/libndi
UltraLeap not found. Make sure LeapC.dll / dylib is in the right path.
...
Player: listening on  40591
...
Warning: Numeric mode unsupported in the posix collation implementation (:0)

jcelerier commented 1 week ago

okay, I can reproduce the crash locally, as well as the drag'n'drop not working with Hyprland. Thanks for the investigation!

ossia built against ArchLinux Qt libraries :

QT_QPA_PLATFORM=xcb : drag'n'drop works
QT_QPA_PLATFORM=xcb : VST gui works, no crash
QT_QPA_PLATFORM=xcb : mouse cursor scales badly when going over the open vst icon
QT_QPA_PLATFORM=xcb : absolutely terrible automatic scaling by the compositor
QT_QPA_PLATFORM=wayland : drag'n'drop works
QT_QPA_PLATFORM=wayland : vst crash with the XCreateWindow error.
- This is caused by X11 not managing to create the window (makes sense as we're running under wayland), which calls XDefaultError which seems to call exit on the whole program, without leaving any chance for recovery.
QT_QPA_PLATFORM=wayland : 2x scaling for no good reason applied by the compositor but at least it's crisp, not pixelated

ossia built against Nix Qt libraries :

QT_QPA_PLATFORM=xcb : drag'n'drop fails
QT_QPA_PLATFORM=xcb : VST gui works, no crash
QT_QPA_PLATFORM=xcb : mouse cursor scales correctly
QT_QPA_PLATFORM=xcb : absolutely terrible automatic scaling by the compositor
QT_QPA_PLATFORM=wayland : drag'n'drop works
QT_QPA_PLATFORM=wayland : vst crash with the XCreateWindow error - same as above, expected

jcelerier commented 1 week ago

Drag'n'drop seems to be broken in Hyprland : https://github.com/hyprwm/Hyprland/issues/1083 Seems to be a solution given here : https://github.com/hyprwm/Hyprland/issues/1969

jcelerier commented 1 week ago

Note that VST2 / VST3 plug-ins GUIs will only ever work with QT_QPA_PLATFORM=xcb, not wayland as the plug-ins themselves expect a X11 host.

jcelerier commented 1 week ago

I will add some checking code that will disable the plug-in GUI button if we happen to be under wayland as there's no way to make this work short of running the entire VST in an external process (with the expected performance cost and implementation complexity)

travis-anderson commented 1 week ago

Drag'n'drop seems to be broken in Hyprland : hyprwm/Hyprland#1083 Seems to be a solution given here : hyprwm/Hyprland#1969

I wonder if this also relates to intermittent dragging issues that I have in Bitwig with hyprland which I didn't have in sway. I'll try the suggested solution. If nothing works ideally, I'll try in sway.

Platform xcb understood. My session-wide $QT_QPA_PLATFORM is actually set to QT_QPA_PLATFORM=wayland;xcb but ossia-score doesn't seem to recognize that as a fallback for the VST.

travis-anderson commented 1 week ago

Btw, I don't have any scaling issues. Using a standard HD monitor (external) and built-in HiDPI (Steam Deck).

jcelerier commented 1 week ago

I don't think any fallback is possible, you have to pass a pointer to the host X11 display when you create a vst. Or if it is possible in any way, I'm very interested in the code that allows it :)

I found this recent bugfix in Qt : https://codereview.qt-project.org/c/qt/qtbase/+/563851 which should fix drag'n'drop. I will update the Qt version score builds against soon.

travis-anderson commented 1 week ago

My experience with C++ is extremely limited, but if I do find a workaround I'll let you know.

jcelerier commented 1 week ago

it would be worth seeing how e.g. qtractor and lmms do it as they both are Qt-based apps able to load plug-ins

jcelerier commented 1 week ago

ikewise for Carla

travis-anderson commented 1 week ago

Well, I loaded Carla in LMMS and was able to open the UI for Dexed there with no issue.

travis-anderson commented 1 week ago

Seems like Carla sets the env var explicitly to support the action https://github.com/falkTX/Carla/blob/main/source/frontend/carla-plugin#L606

This could probably be handled within the nix package without changing any of the app code.

jcelerier commented 1 week ago

well, that's what I do in ossia too.. so I don't know why you ended up having ossia start with wayland:

https://github.com/ossia/score/blob/e0676c9fe340df3975ccde2a098cfa15610fff3f/src/app/main.cpp#L116

this function forces QT_QPA_PLATFORM to xcb unless the user did set it explicitly : https://github.com/ossia/score/blob/e0676c9fe340df3975ccde2a098cfa15610fff3f/src/app/main.cpp#L148

travis-anderson commented 1 week ago

Ah, yeah, this variable is set for my session, so it wouldn't override it. I'm rebuilding with the following added to the nix pkg I have locally

  postInstall = ''
    wrapProgram "$out/bin/ossia-score" --set QT_QPA_PLATFORM "xcb"
  '';

This should let me run it no matter what I have set for the rest of the session.

travis-anderson commented 1 week ago

I'm happy to report that adding the postInstall worked for ensuring the proper platform, and that adding windowrulev2 = "noinitialfocus,xwayland:1" to my hyprland config seems to have fixed the dragging issues in both ossia-score and Bitwig!

Thank you very much for digging into this with me. I can make a PR to the nix package if you think that's worth it, too.

for any future searchers: the hyprland mouse fix is declared like so if you're using home-manager:

users.*.wayland.windowManager.hyprland.settings.windowrulev2 = [ "noinitialfocus,xwayland:1" ];

to force noinitialfocus for xwayland apps.

jcelerier commented 1 week ago

great :) a PR to the nix package would be awesome !

jcelerier commented 1 week ago

on ossia side I added detection of wayland platform plug-in to disable the VST / VST3 UI so that at least it won't cause crashes to the user

ossia / score

Crash while opening VST instrument plugin UI #1551

ossia built against ArchLinux Qt libraries :

ossia built against Nix Qt libraries :