VST freezes on Arch

[mateuszmnm]

Steps to reproduce:

• start LMMS
• drag VeSTige into main screen
• click on VeSTige's green folder icon
• select a .DLL VST plugin

  Expected:

• loading plugin window appears
• plugin loads

  Actual:

• loading plugin window appears
• one of the CPU cores goes 100%
• LMMS stops responding
• it stays like that forever

Screenshot: http://i.imgur.com/2f5YURI.png Affected system: Arch 64 bit LMMS versions: 1.1.1, 1.1.2, 1.1.3 Wine versions used: at least 4 different - 1.6.4. 1.7.22, 1.7.33, 1.7.38 Example VST: Synth1 More details: https://lmms.io/forum/viewtopic.php?f=7&t=1749

Some strace dump: https://gist.github.com/obszczymucha/161b9fb25ce0a48b6af3 (it goes like that forever)

[obszczymucha]

Doh, posted above on wrong account...

[obszczymucha]

I've profiled LMMS with Valgrind during reproducing the issue and here's the result: https://www.dropbox.com/s/46p9g4bnknvs272/callgrind.out.2681?dl=0

I still don't know what to make out of this, but maybe someone else beats me to it.

[obszczymucha]

If that helps, I've managed to debug LMMS into a point where execution stops at: https://github.com/LMMS/lmms/blob/stable-1.1/plugins/vst_base/VstPlugin.cpp#L180

which is waitForInitDone()

I'm guessing LMMS waits for a signal from remote plugin and either doesn't get it or gets it but doesn't process it properly (or locked?).

Additionally I can see that wineserver process starts by doing 'ps aux'.

[badosu]

Great bug report @obszczymucha! :+1:

I am also experiencing this bug.

[obszczymucha]

Bad news. We are stuck in the loop: https://github.com/LMMS/lmms/blob/stable-1.1/include/RemotePlugin.h#L980

I've added:

printf("_busy_waiting: %d, messagesLeft(): %d\n", _busy_waiting, messagesLeft());

And it always outputs: _busy_waiting: 1, messagesLeft(): 0

[tresf]

Possible duplicate of #1567. Related: #1097 #1736 #607

[badosu]

Reproduced on my machine as well, can't figure out why this happens.

[tresf]

@badosu does this happen with other VSTs (i.e. DSK musicbox?)

[badosu]

Yep, tested on lots of other VSTs. This is not a particular VST problem but with the messagery between wine and lmms as indicated above. At least, it's what it looks like/

[qnebra]

I have this same issue on lmms 1.1.3 from kxstudio in vivid.

[tresf]

@badosu is this a valid bug with our code, or a problem with the way these downstream packages are created?

[badosu]

@tresf I'd have to try a previous release to test this, I really don't know. Gonna take a look.

[mikesbytes]

I can also produce this issue on Arch using the provided packages. This also happens when I use the git version from AUR.

[Gnurou]

Same issue, also with Arch Linux 64 bit. Will try to investigate a bit further if I can find the time.

[Umcaruje]

Can any of you confirm this bug on Arch if you manually compile LMMS and not use the packages?

[obszczymucha]

I've built multiple versions from the source. Same problem. I'll try and build the latest 1.1.3 version and let you know the outcome.

[obszczymucha]

Just built 1.1.3 - the issue is still there.

[Gnurou]

Yes, this is from a source build. I have started looking at it, but could not rootcause it yet, excepted that it seems to be a miscommunication between the main process and RemoteVstPlugin. That part of the code is not particularly easy to comprehend to new eyes. Here is the top of the backtrace from the main process:

#0  0x00007ffff7bcdc1d in recvmsg () from /usr/lib/libpthread.so.0
#1  0x00007ffff082e917 in ?? () from /usr/lib/libxcb.so.1
#2  0x00007ffff082f188 in ?? () from /usr/lib/libxcb.so.1
#3  0x00007ffff28abdc8 in ?? () from /usr/lib/libX11.so.6
#4  0x00007ffff28abf2b in ?? () from /usr/lib/libX11.so.6
#5  0x00007ffff28ac23d in _XEventsQueued () from /usr/lib/libX11.so.6
#6  0x00007ffff289db10 in XEventsQueued () from /usr/lib/libX11.so.6
#7  0x00007ffff713bbf7 in ?? () from /usr/lib/libQtGui.so.4
#8  0x00007ffff3fcb1bd in g_main_context_prepare () from /usr/lib/libglib-2.0.so.0
#9  0x00007ffff3fcbba8 in ?? () from /usr/lib/libglib-2.0.so.0
#10 0x00007ffff3fcbd8c in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
#11 0x00007ffff6952044 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQtCore.so.4
#12 0x00007ffff713c156 in ?? () from /usr/lib/libQtGui.so.4
#13 0x00007ffff69255ce in QCoreApplication::processEvents(QFlags<QEventLoop::ProcessEventsFlag>, int) () from /usr/lib/libQtCore.so.4
#14 0x000000000053607e in RemotePluginBase::waitForMessage (this=0x1b6ae00, _wm=..., _busy_waiting=true) at /home/gnurou/Projects/lmms/lmms/include/RemotePlugin.h:954
#15 0x0000000000537fc7 in RemotePlugin::waitForInitDone (this=0x1b6ae00, _busyWaiting=true) at /home/gnurou/Projects/lmms/lmms/include/RemotePlugin.h:690
#16 0x00007fffa2732d19 in VstPlugin::tryLoad (this=0x1b6add0, remoteVstPluginExecutable=...) at /home/gnurou/Projects/lmms/lmms/plugins/vst_base/VstPlugin.cpp:183
#17 0x00007fffa27327dc in VstPlugin::VstPlugin (this=0x1b6add0, _plugin=...) at /home/gnurou/Projects/lmms/lmms/plugins/vst_base/VstPlugin.cpp:102
#18 0x00007fffa1bf4f51 in vestigeInstrument::loadFile (this=0x7fffeafb7da0, _file=...) at /home/gnurou/Projects/lmms/lmms/plugins/vestige/vestige.cpp:260
#19 0x00007fffa1bf73f8 in VestigeInstrumentView::openPlugin (this=0x18d8440) at /home/gnurou/Projects/lmms/lmms/plugins/vestige/vestige.cpp:650
#20 0x00007fffa1bfd178 in VestigeInstrumentView::qt_static_metacall (_o=0x18d8440, 
_c=QMetaObject::InvokeMetaMethod, _id=1, _a=0x7fffffffd060) at /home/gnurou/Projects/lmms/lmms/build/plugins/vestige/moc_vestige.cxx:217

What is weird is that it seems to happen only on Arch Linux?

Let me know if I can provide more information.

[hydrix9]

confirming from arch x64 with latest versions of wine and lmms

[Narfinger]

I think I tracked down the wait to RemotePlugin.h:293. We are seeming to wait on a semaphore.

Interesting enough, I tried to change to define USE_QT_SEMAPHORES which still crashes lmms in the same way but it gives me the GUI of the VST.

[tresf]

I'd love to reproduce this problem and help troubleshoot, but I simply don't have the patience for a 50 page wiki article to get the OS loaded in a VM and I think this is an unreasonable request for the development team of music software.

Can someone provide a working VM image? By "working", I mean something I can boot and use that is pre-partitioned, has networking installed, a GUI, a boot loader, et. al?

I've skimmed a few quickstart guides, and even the "quick start" guides seem to take hours to complete, which is hours that could be better spent trying to fix this bug.

-Tres

[Wallacoloo]

@tresf you could try one of the Arch-based distros.

And if you're running it in a VM you shouldn't need to do any boot loader stuff.

[tresf]

@tresf you could try one of the Arch-based distros.

Tremedous. I haven't heard of or tried any of those. Any recommendation?

[tresf]

Manjaro Linux seems to be #\8 on distrowatch. I'll check that one out.

Edit: Installed Manjaro, compiling.

sudo pacman -S wine cmake qt4 libsndfile fftw libvorbis libogg alsa-plugins jack\
sdl libsamplerate stk fluidsynth portaudio ftlk libxinerama libxft libgig git gcc-multilib\
lib32-qt4

[Narfinger]

Another thing which I noticed in a debug session is that RemotePluginBase::waitForMessage is busy waiting while zero messages are left. Perhaps isInvalid() is not updated correctly?

[tresf]

Sorry I hadn't yet posted an update on this... My Manjaro install suffers the exact same symptoms of that in the original bug report. shmget: No such file or directory

I think the long term solution is to add better exception handling to our RemotePlugin code. When LMMS talks to an external process (this happens for VST(fx), VST(i) and AFAIK ZynAddSubFX's GUI uses it too).

In the short term, the major troubleshooting issue we have is in these scenarios where our shared memory logic fails with no sign of where or why. Here's an article that explains (perhaps) how we can do better debugging: http://stackoverflow.com/a/7497455/3196753

I believe this is in part due to the choice to use native C++ shared memory for some platforms, but Qt Shared memory for others. I'm making some broad assumptions, but I assume shared memory started as a *nix-Only feature and matured into using Qt's more platform independent implementation for the proprietary platforms, -- such as Windows (and eventually for Mac #698, #703 -- once we get that sorted out)

https://github.com/LMMS/lmms/issues/1468#issuecomment-72375290

Although I'm curious if we should just switch our code to use Qt for all platforms, since it comes as a dependency on our codebase anyway. AFAIK, Mac -- for example -- has extremely low SHMEM thresholds when using pure C++. This requires applications to roll their own SHMEM calls. Qt should help circumvent these limitations.

The only person on the project I've ever seen actually understand this remote process complexity is the author, (and the original author of LMMS), @tobydox. Perhaps we'll get lucky and get some insight from him as to how to fix these types of things, or at a minimum, how to debug them.

Another option is to make some more test scenarios and allow Travis-CI to do the RemotePlugin testing for platforms such as Apple, since we have Travis-CI support for that platform.

[mkondel]

The shmget error goes away after installing wine1.7, at least on ubuntu. The VSTs still do not load, even after compiling the latest master. It's not only Arch where this happens.

[tresf]

I believe this is a duplicate of forums/#1640 and possibly #2196.

FYI, Here's Toby's response from January 2014 on the topic...

Tobias Doerffel 2014-01-23 09:52:53 CST

Hi,

as the original main developer of LMMS and its VST support layer, I'm joining. First of all thanks for your efforts to fix this issue! Regarding your questions:

The source code of the application which is launched externally using WINE can be found here:

https://github.com/LMMS/lmms/blob/stable-0.4/plugins/vst_base/RemoteVstPlugin.cpp

At line 692 you can find the code which retrieves the X11 window ID. This code always has been working. Is there any other recommended method?

Inside LMMS we use QX11EmbedContainer to embed the window inside our MDI sub window:

https://github.com/LMMS/lmms/blob/stable-0.4/plugins/vst_base/VstPlugin.cpp

at line 245 ff. So basically we don't do any magic here.

-Tres

[tresf]

Some more background information:

https://github.com/LMMS/lmms/issues/1468#issuecomment-71485618

[mkondel]

I get a message like this on the console when VST is stuck in the infinite loop:

fixme:ole:RemUnknown_QueryInterface No interface for iid {00000019-0000-0000-c000-000000000046}

Searching on google shows lots of Wine related issues, but the fixes are all over the place, these cant help with VST... Ex:

- After the latest update [of Wine prolly], my tax return software works
- installing internet explorer in the same wineprefix solved it for me
- the problem was that I didn't have lib32-nvidia-libgl

[tresf]

Ok, I found something that wasn't in any of our forums or bug reports.... An upstream bug that was filed with Wine and claims to be fixed with Wine 1.7.11. Not sure if this is related, but it has the same copy/paste message that came from Toby in the forums...

https://bugs.winehq.org/show_bug.cgi?id=35347 https://bugreports.qt.io/browse/QTBUG-36446

Wine claims to have patched this in version 1.7.11 (My Manjaro install is using Wine 1.7.49), but there is a recommendation in there which we may find useful as it describes that if Qt eventually fixes this, our code can cause issues.

If the problems with QT are fixed later you will probably experience an additional problem, which is caused by the fact that you make the window visible before its embedded. This never works that well, and I would recommend to do this after the reparenting step (by using some additional ipc messages).

Sorry if this is all inaccurate such as in the event this bug is actually fixed. Trying to dig up more info on this...

So our RemotePlugin seems like it aims to do 3 things:

1. Start a remote process
2. Embed that remote process as a Qt sub-window
3. Share memory with the remote process for data exchange

We may need to write a sample C++ test program to simulate this in order to know how to debug where something is breaking.

-Tres

[mkondel]

If RemoteVstPlugin.cpp is the app that runs in Wine, would that be a good starting point to debug what's breaking? I seem to get it to printf to the console, at least getting to line 357 at the moment.

[tresf]

If RemoteVstPlugin.cpp is the app that runs in Wine, would that be a good starting point to debug what's breaking?

Sure, just take note that RemotePlugin code is getting executed here as well.

[mkondel]

The execution seems to hang here: https://github.com/LMMS/lmms/blob/master/plugins/vst_base/RemoteVstPlugin.cpp#L407 Line 407 gets called, but the following code, in RemotePlugin, is never run: https://github.com/LMMS/lmms/blob/master/include/RemotePlugin.h#L922

[Narfinger]

I removed the non-qt ShmMem and Semaphore code. It can be found in master of my repository https://github.com/Narfinger/lmms on master.

However, now I get a wine backtrace which I do not know how to debug.

[tresf]

@Narfinger do to this proper, should you tackle RemotePlugin as well?

With your branch, I'm getting this on Ubuntu 12.04. I'm guessing because some of the dependencies aren't in order for non-Windows QSharedMemory usage.

[ 81%] Generating RemoteVstPlugin
In file included from /home/tres/lmms/plugins/vst_base/RemoteVstPlugin.cpp:35:0:
/home/tres/lmms/include/RemotePlugin.h:41:27: fatal error: QtCore/QtGlobal: No such file or directory
compilation terminated.
winegcc: g++ failed
make[2]: *** [plugins/vst_base/RemoteVstPlugin] Error 2
make[1]: *** [plugins/vst_base/CMakeFiles/vstbase.dir/all] Error 2
make: *** [all] Error 2
tres@ubuntu-1204:~/lmms/build$ vi /home/tres/lmms/plugins/vst_base/RemoteVstPlugin.cpp
tres@ubuntu-1204:~/lmms/build$ 

-Tres

[Narfinger]

Hey. I just updated my branch, indeed I forgot something. I also had to change some cmake file. This was rather ad-hoc.

But good news. Synth1 loads for me. If it loads for other people (somebody should probably test this on windows) I can cleanup the changes and do a pull request.

[mkondel]

Any quick ideas? I tried to build your fork on master:

Scanning dependencies of target vibedstrings
[ 44%] Building CXX object plugins/vibed/CMakeFiles/vibedstrings.dir/vibed.cpp.o
/usr/bin/ld: cannot find -lQtCore
collect2: error: ld returned 1 exit status
winegcc: g++ failed
plugins/vst_base/CMakeFiles/vstbase.dir/build.make:53: recipe for target 'plugins/vst_base/RemoteVstPlugin' failed
make[2]: *** [plugins/vst_base/RemoteVstPlugin] Error 2
CMakeFiles/Makefile2:8539: recipe for target 'plugins/vst_base/CMakeFiles/vstbase.dir/all' failed
make[1]: *** [plugins/vst_base/CMakeFiles/vstbase.dir/all] Error 2

[tresf]

If it loads for other people (somebody should probably test this on windows) I can cleanup the changes and do a pull request.

A pull request will do the Windows testing for you automatically due to our Travis-CI integration. :+1:

[tresf]

Any quick ideas? I tried to build your fork on master:

I'm not having any luck from Ubuntu side either... A pull request will show the errors for Ubuntu in Travis-CI too but in my case it might be a 32/64bit lib thing. I'm running a clean build now to capture the exact error.

[tresf]

Here's my error (Ubuntu 12.04) which is the same thing @mkondel is receiving, just more verbose.

Also @mkondel don't use -j2 or -j4 with make when you're posting a debug message as you'll get success messages mixed into error messages, as are your "vibedstrings" lines.

At a glance, I think the -m32 switch wants the 32-bit libpthread.so perhaps? Sorry if I'm of no assistance.

[ 81%] Generating RemoteVstPlugin
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libpthread.so when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libpthread.a when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libQtCore.so when searching for -lQtCore
/usr/bin/ld: cannot find -lQtCore
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libwine.so when searching for -lwine
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libc.so when searching for -lc
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libc.a when searching for -lc

[Narfinger]

This happens because I hacked the Cmake file to work for my system but did not check how to do this in general because I didn't have time. It should be fixed in the pull request I will do later.

[ghost]

@Narfinger

I have ArchLinux 64bit too, and your fork with the latest commit finally allows me to run 32bit Synth1 successfully, however the 64bit version just says "remote plugin died, invalidating now".

(Oh BTW, to compile lmms, install gcc-multilib and lib32-qt4 packages)

[Narfinger]

I don't know about 64 bit, did 64 bit plugins ever work? It doesn't look like it.

For the bugs above: I think @ichiman94 is right, you need the 32bit qt4 packages (at least QtCore). I have a fixed-vestige branch on my repository, however there is a slight problem: At the moment the -lQtCore flag is hardcoded in the cmake file as find_package in cmake will just find the 64 bit thing but we need to explicitly link against the 32 bit libraries.

I at the moment do not know how to have a good cmake file. Somebody from the cmake channel said we should look into ExternalProject and make this a 32 bit project (because we can only make a whole cmake project into 32bit). But I need to read up on this.

[tresf]

At the moment the -lQtCore flag is hardcoded in the cmake file as find_package in cmake will just find the 64 bit thing but we need to explicitly link against the 32 bit libraries.

According to this mailing list archive -m32 should be enough. http://www.cmake.org/pipermail/cmake/2013-February/053595.html

Although I have feeling these paths are borrowed from qmake rather than cmake so some other type of trickery may be needed.

Optimistically, this posts suggests it might just be a missing flag...

ADD_LIBRARY(mylib SHARED my_source.c)
SET_TARGET_PROPERTIES(mylib PROPERTIES COMPILE_FLAGS "-m32" LINK_FLAGS "-m32")

http://stackoverflow.com/a/19310455/3196753

Edit: P.S. Great collaborative work going on in this thread. :+1:

[Zeioth]

I'm experiencing exactly the same problem under Xubuntu 15.04 x64 + wine-1.7.50 + LMMS 1.1.3 from kxstudio. I'm glad to hear the solution it's on the way, I'll keep in the loop.

More system info -> https://i.imgur.com/BLCu4wy.png

[Wallacoloo]

@tresf for future testing on arch (rather than a derivative like Manjaro), I came across an unofficial arch installer that you may find useful: https://sourceforge.net/projects/dh-arch-installer/

[tresf]

Install arch-linux in a damn hurry

lol.

@Wallacoloo is this recommended over Manjaro? I don't mind either approach, BTW.

[Wallacoloo]

@tresf you'll be using identical base system, repositories and packages as a normal arch system (I would imagine Manjaro applies some branding and/or patching to the normal arch packages), which means you should have more success in replicating any arch-only bugs.

But if you already have Manjaro up and running, then no need to try this method unless you ever fail to reproduce an arch-only bug.

[tresf]

:+1: