Tracking issue for wrapQtAppsHook

ttuegel commented 5 years ago

This issue is to track the open pull requests for Qt applications broken by the addition of wrapQtAppsHook.

Maintainers, please refer to the manual for instructions to update your packages. Packages that conformed to the prior version of the manual should not need to be updated.

Pull requests

65366
65372

ambrop72 commented 5 years ago

I want to fix #65564 but I still don't understand exactly what changed that caused VirtualBox to break. What exactly was changed in the behavior of the Qt derivations which results VirtualBox no longer finding what it needs? Was the default search behavior of Qt somehow modified?

samueldr commented 5 years ago

The following explanation might lack precision, but the gist of it is basically right.

"Nothing" changed in how Qt looks for its libraries, as far as Qt is concerned. What changed in #54525 is that with the change, it relies on adding wrappers to correctly find the right dependencies.

Those wrappers, in turn, AFAIUI, breaks assumptions in the hardening.

ambrop72 commented 5 years ago

@samueldr The interesting thing is that the VirtualBoxVM binary, when used with hardening, always unsets QT_PLUGIN_PATH. So the reason that it worked before cannot be that QT_PLUGIN_PATH happened to include the right paths. So just by what mechanism did it find the plugins before? Maybe something related to propagatedBuildInputs?

Though, it really does not matter any more because I've just managed to make it work and will open a PR soon.

ambrop72 commented 5 years ago

I meant propagatedUserEnvPkgs...

ttuegel commented 5 years ago

So the reason that it worked before cannot be that QT_PLUGIN_PATH happened to include the right paths. So just by what mechanism did it find the plugins before?

We have a custom patch which scans for plugins relative to entries in PATH, and this was probably how it worked. This mechanism has always been very fragile; the purpose of the changes made to Qt packaging was to avoid relying on this mechanism. I don't have an explanation for why this mechanism has broken now, though.

ambrop72 commented 5 years ago

So the reason that it worked before cannot be that QT_PLUGIN_PATH happened to include the right paths. So just by what mechanism did it find the plugins before?

We have a custom patch which scans for plugins relative to entries in PATH, and this was probably how it worked. This mechanism has always been very fragile; the purpose of the changes made to Qt packaging was to avoid relying on this mechanism. I don't have an explanation for why this mechanism has broken now, though.

I think this should be removed. One problem is that it is a security issue for VirtualBox which is setuid (not sure exactly when it drops privileges though).

ttuegel commented 5 years ago

I think this should be removed.

So do I, that was the primary motivation introducing wrapQtAppsHook.

leenaars commented 5 years ago

Is there any guidance how to do this with PyQt based applications? See https://github.com/NixOS/nixpkgs/pull/68405

acowley commented 5 years ago

I am not certain which parts of this report are related to this issue: I followed the nixos-unstable channel to e19054ab3cd5b7cc9a01d0efc71c8fe310541065

I have kdenlive among my systemPackages; after the update, running kdenlive from a Konsole window results in hundreds of warnings and "Unknown asset" lines followed by a segfault. I can launch kdenlive from the menu. It starts and seems to work, but rendering a video fails with this error log:

Rendering of /home/acowley/Videos/bloop.mp4 crashed

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem. Available platform plugins are: wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, minimal, offscreen, vnc, xcb, wayland-org.kde.kwin.qpa.

Usually I start kdenlive from the terminal so I see all its gory log output, but I don't know how to reveal that when starting it from the launcher. This latter error looks like a helper executable not being wrapped, but I do not know which executable that might be.

bjornfor commented 5 years ago

@acowley: sounds related to https://github.com/NixOS/nixpkgs/issues/68404.

acowley commented 5 years ago

It seems like mlt might not be properly wrapped, which could be the rendering problem. No idea yet on the launching from a terminal problem.

Edit: I made the suggested changes to the mlt/qt5.nix expression (taking mkDerivation as a parameter), but nothing is changed: kdenlive segfaults when started from a terminal; starts okay from the launcher, but rendering fails with the same error message as above.

Edit 2: It turned out I had konsole toggled with "Pin to Task Manager" on the desktop. This created a multi-QT environment as the pinned version was stale; that was the cause of kdenlive segfaulting when launched from the terminal. Rendering crash still under investigation.

acowley commented 5 years ago

I think @bjornfor was correct that the melt executable is not properly wrapped, as I was able to follow the suggestion at his link and end up with a better-looking wrapper. However, it turns out this was not the cause of the error I saw. kdenlive caches paths for an MLT profiles folder and a Melt path under Settings -> Configure Kdenlive -> Environment. If you clear these out, it will discover the right paths and populate the GUI elements. I'm working on a patch to get us out of that hole.

tobiasBora commented 5 years ago

I'm just curious, if I understand, all apps should use mkDerivation instead of stdenv.mkDerivation no? Then, why can't we "just" search all stdenv.mkDerivation and turn them into mkDerivation, at least for simple enough packages? That could avoid bad surprises at runtime.

worldofpeace commented 5 years ago

@tobiasBora mkDerivaiton here is from qt*. Everyone should use stdenv.mkDerivation unless it's a qt application and needs that special deriver.

Though it was discussed if we could get a semi-automated way to figure out what expressions need to be fixed.

ttuegel commented 5 years ago

Though it was discussed if we could get a semi-automated way to figure out what expressions need to be fixed.

It would be simple, at least, to add a check to the qtbase setup hook that some special variable is set by the deriver, and fail those builds with do not use it.

worldofpeace commented 5 years ago

@ttuegel That actually does sound very simple :sparkles:

If we could set that up we could get a hydra jobset for the PR and I think have them all fixed. I'm not sure at this point we could block 19.09 on that, we have to release eventually. But all the fixes that come out of that should be completely fine for backports.

nh2 commented 5 years ago

For meshlab, I'm moving to the hook in #70937.

FRidh commented 5 years ago

Is there really no way we can do without a custom deriver? I mean, Qt in the end are "just" libraries, its not its own build system. I get there are issues, but, using custom derivers prevents composition.

ttuegel commented 5 years ago

Is there really no way we can do without a custom deriver? I mean, Qt in the end are "just" libraries, its not its own build system.

@FRidh I think Qt is closer to Python or Emacs because it (effectively) has its own runtime system. It's not just a library on top of the C runtime. That aside, your point is well taken, and I will see about getting rid of the custom deriver.

f--t commented 5 years ago

@ttuegel It seems quite a few (perhaps all) of the python packages depending on pyqt5 an pyqt4 are currently broken, e.g. matplotlib qt backend, jupyter qt-console.

Since python packages don't directly use stdenv, the doc referenced above doesn't directly apply. Are there any suggestions on the idiomatic way fixing this. Using propagatedBuildInputs with wrapQtAppsHook in pyqt5 doesn't fix the problem.

rnhmjoj commented 4 years ago

Could someone finish the review at #67806?

KoviRobi commented 4 years ago

Not sure this is something we care about too much, but sometimes python+Qt packages seem to be double-wrapped. See openshot-qt and https://github.com/NixOS/nixpkgs/pull/71335. Is there a way to avoid this (seems like rapid-photo-downloader avoids it)? But it's also a hand-coded one, not sure if there should be a library function for it.

worldofpeace commented 4 years ago

You can do

dontWrapQtApps = true;
makeWrapperArgs = [
  "\${qtWrapperArgs[@]}"
];

I've documented to do the something like this for gtk applications recently

https://nixos.org/nixpkgs/manual/#ssec-gnome-common-issues

edef1c commented 4 years ago

I've needed to use "\"\${qtWrapperArgs[@]}\"", which I assume is what you meant. I've been saving some PRs for after I figure out how to get rid of that. Seems like an escaping issue we should be able to fix.

worldofpeace commented 4 years ago

I've needed to use "\"\${qtWrapperArgs[@]}\"", which I assume is what you meant. I've been saving some PRs for after I figure out how to get rid of that. Seems like an escaping issue we should be able to fix.

Oops, guess I misspelled. Updated the example. I've actually suggested this a while ago so there's examples in nixpkgs that work fine

https://github.com/NixOS/nixpkgs/blob/c9b85da380d4c15e3bec8b7d2124455e0fbae423/pkgs/applications/networking/browsers/webmacs/default.nix#L32

Then afterwards documented it for gtk.

edef1c commented 4 years ago

Basically, this approach seems to break with GTK, because the array may contain empty elements (for environment variables that don't actually need prefixing with anything). Improving that (by excluding empty vars from being appended) causes a rather huge rebuild, so I decided against it. Frankly, I think the fix for that belongs in buildPythonApplication, since we shouldn't be doing word splitting on makeWrapperArgs anyway.

laikq commented 4 years ago

Hi, are there any ideas or progress on how to fix pyqt -> matplotlib & others?

wucke13 commented 4 years ago

Some that are yielding the good ol' wrapQtAppsHook:

andersk commented 4 years ago

I opened #75405 two months ago to fix muse, but aside from some nitpicking over whether to use Qt mkDerivation or wrapQtAppsHook, it seems to have been ignored and muse is still broken. Meanwhile, other PRs are being merged in both styles. Is one of mkDerivation or wrapQtAppsHook preferred over the other?

worldofpeace commented 4 years ago

I opened #75405 two months ago to fix muse, but aside from some nitpicking over whether to use Qt mkDerivation or wrapQtAppsHook, it seems to have been ignored and muse is still broken. Meanwhile, other PRs are being merged in both styles. Is one of mkDerivation or wrapQtAppsHook preferred over the other?

I merged it. It's not that big of a deal as long as it's fixed. In general we should just do what the docs say.

wucke13 commented 4 years ago

paraview needs the hook too

lapp0 commented 4 years ago

tikzit needs the hook as well

wucke13 commented 4 years ago

qview needs a fix

bbarker commented 4 years ago

qgo appears to need this

worldofpeace commented 4 years ago

Various packages that were mentioned needed fixes are in https://github.com/NixOS/nixpkgs/pull/80982

rnhmjoj commented 4 years ago

If anyone has any idea what to do about matplotlib and python libraries... I would be happy to try implementing it. The issue is https://github.com/NixOS/nixpkgs/issues/80147

yshym commented 4 years ago

android-studio (avd) seems to need this

9:29 PM Emulator: Fatal: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

9:29 PM Emulator: 

9:29 PM Emulator: Available platform plugins are: xcb.

9:29 PM Emulator: ((null):0, (null))

9:29 PM Emulator: Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

nixos-discourse commented 4 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/please-fix-all-packages-which-were-broken-by-the-qt-wrapping-changes/6444/16

unode commented 4 years ago

mnemosyne is also failing. Since it's a python application that uses pyqtwebengine and python.pkgs.buildPythonApplication, what is the recommendation for a fix here?

rnhmjoj commented 4 years ago

@unode: Take a look at expression for qutebrowser.

unode commented 4 years ago

Thanks @rnhmjoj . Pull request #84954

shuhaowu commented 4 years ago

I think this wrap with the python development mode is broken. See this demo repository I created: https://github.com/shuhaowu/nix-qt-python-dev-mode-failure.

To reproduce:

Clone that repository.
$ nix-shell
$ qt-test

Watch the following error occur and the program crashes:

qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in ""
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

However, outside of nix-shell is fine. To reproduce:

Clone that repository.
$ nix-build
$ result/bin/qt-test
See the hello world message.

What's going on here? Also any criticism on my default.nix file is welcome as I'm still a nix noob.

shuhaowu commented 4 years ago

The previous issue seems to be solved if the wrapQtApp wrapped script's environment variable is exported into command line. For me this was:

export QT_PLUGIN_PATH='/nix/store/0d8hv33sqxbzi3gkvcv1wd7y101n9ldv-qtbase-5.12.7-bin/lib/qt-5.12.7/plugins'${QT_PLUGIN_PATH:+':'}$QT_PLUGIN_PATH
export QT_PLUGIN_PATH='/nix/store/a8b5vd0hy86pqmrsvxdx88qzpd4kkk09-qtsvg-5.12.7-bin/lib/qt-5.12.7/plugins'${QT_PLUGIN_PATH:+':'}$QT_PLUGIN_PATH
export QT_PLUGIN_PATH='/nix/store/jjyjr2ck1k21s6z7lr3ip2yj7d10k4iw-qtdeclarative-5.12.7-bin/lib/qt-5.12.7/plugins'${QT_PLUGIN_PATH:+':'}$QT_PLUGIN_PATH
export QML2_IMPORT_PATH='/nix/store/jjyjr2ck1k21s6z7lr3ip2yj7d10k4iw-qtdeclarative-5.12.7-bin/lib/qt-5.12.7/qml'${QML2_IMPORT_PATH:+':'}$QML2_IMPORT_PATH
export QML2_IMPORT_PATH='/nix/store/gbal6xsngnpsdbw40qqvfmddrqb8ck73-qtquickcontrols-5.12.7/lib/qt-5.12.7/qml'${QML2_IMPORT_PATH:+':'}$QML2_IMPORT_PATH
export QT_PLUGIN_PATH='/nix/store/j86x96jnfjk6f7p63kfm9q5c19c2dc9c-qtwayland-5.12.7-bin/lib/qt-5.12.7/plugins'${QT_PLUGIN_PATH:+':'}$QT_PLUGIN_PATH
export QML2_IMPORT_PATH='/nix/store/j86x96jnfjk6f7p63kfm9q5c19c2dc9c-qtwayland-5.12.7-bin/lib/qt-5.12.7/qml'${QML2_IMPORT_PATH:+':'}$QML2_IMPORT_PATH

Adding this before launching qt-test from nix-shell appears to work! Now to turn this into a nix-shell friendly solution is a different problem...

stale[bot] commented 4 years ago

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

worldofpeace commented 4 years ago

I believe this is like 99% no longer an issue.

Xe commented 3 years ago

I'm encountering this with plover on NixOS 20.09:

$ nix-env -iA nixos.plover.dev
installing 'python3.6-plover-4.0.0.dev8'
these paths will be fetched (26.60 MiB download, 149.08 MiB unpacked):
  /nix/store/1c86mzzxdn1gh3zsr8m48pf31vqwk92p-python3.6-PyQt5.sip-4.19.24
  /nix/store/1yzd3q817p1kznaxh5nqwgbqqk24pdhc-python3.6-dbus-python-1.2.16
  /nix/store/50bbqdln7l97b36awlw019r25bl3f8xw-python3.6-appdirs-1.4.4
  /nix/store/6j31qc3nv1znmp98bvbcrydy5bikrpwc-python3.6-pytz-2020.1
  /nix/store/9hw5qi4q2gsw7gjyb1z9mz0qa4szf20f-python3.6-six-1.15.0
  /nix/store/bb47dgrdx1qgyw5zjvlmy503g2yj6b0b-python3.6-pyserial-3.4
  /nix/store/bjncdzcpgvv52sbq0qr8c6aa3lk1basa-python3.6-xlib-0.25
  /nix/store/bl7cj3fxgr43qwy48w96xrbxrqgi89a3-python3.6-dbus-python-1.2.16-dev
  /nix/store/d3lxh9n89rsf28v5zyn8pbghbc0cwqmg-python3.6-PyQt5-5.15.1
  /nix/store/hss1wvb697a1wwnakbxsqm6q14wd2pb5-python3.6-plover-4.0.0.dev8
  /nix/store/im3nhrxmbmdszffm56p64d938hvhl7im-python3.6-wcwidth-0.2.5
  /nix/store/iswawhahkrdvwcwkr9qkkf36v7rxgfm4-python3.6-PyQt5-5.15.1-dev
  /nix/store/l0pzd5jxgg6hj23si7kf3cp9cpkj3r96-python3-3.6.12
  /nix/store/whxcnpcdi8rvwgx1cg794ximylvf38rr-python3.6-setuptools-47.3.1
  /nix/store/x6fxich0cyszdiq34zh7jwl2470xk57k-python3.6-Babel-2.7.0
copying path '/nix/store/l0pzd5jxgg6hj23si7kf3cp9cpkj3r96-python3-3.6.12' from 'https://cache.nixos.org'...
copying path '/nix/store/1yzd3q817p1kznaxh5nqwgbqqk24pdhc-python3.6-dbus-python-1.2.16' from 'https://cache.nixos.org'...
copying path '/nix/store/50bbqdln7l97b36awlw019r25bl3f8xw-python3.6-appdirs-1.4.4' from 'https://cache.nixos.org'...
copying path '/nix/store/bb47dgrdx1qgyw5zjvlmy503g2yj6b0b-python3.6-pyserial-3.4' from 'https://cache.nixos.org'...
copying path '/nix/store/whxcnpcdi8rvwgx1cg794ximylvf38rr-python3.6-setuptools-47.3.1' from 'https://cache.nixos.org'...
copying path '/nix/store/6j31qc3nv1znmp98bvbcrydy5bikrpwc-python3.6-pytz-2020.1' from 'https://cache.nixos.org'...
copying path '/nix/store/9hw5qi4q2gsw7gjyb1z9mz0qa4szf20f-python3.6-six-1.15.0' from 'https://cache.nixos.org'...
copying path '/nix/store/bl7cj3fxgr43qwy48w96xrbxrqgi89a3-python3.6-dbus-python-1.2.16-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/1c86mzzxdn1gh3zsr8m48pf31vqwk92p-python3.6-PyQt5.sip-4.19.24' from 'https://cache.nixos.org'...
copying path '/nix/store/bjncdzcpgvv52sbq0qr8c6aa3lk1basa-python3.6-xlib-0.25' from 'https://cache.nixos.org'...
copying path '/nix/store/x6fxich0cyszdiq34zh7jwl2470xk57k-python3.6-Babel-2.7.0' from 'https://cache.nixos.org'...
copying path '/nix/store/d3lxh9n89rsf28v5zyn8pbghbc0cwqmg-python3.6-PyQt5-5.15.1' from 'https://cache.nixos.org'...
copying path '/nix/store/im3nhrxmbmdszffm56p64d938hvhl7im-python3.6-wcwidth-0.2.5' from 'https://cache.nixos.org'...
copying path '/nix/store/iswawhahkrdvwcwkr9qkkf36v7rxgfm4-python3.6-PyQt5-5.15.1-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/hss1wvb697a1wwnakbxsqm6q14wd2pb5-python3.6-plover-4.0.0.dev8' from 'https://cache.nixos.org'...
building '/nix/store/lyy5m5m81iygkxcmh9vgb74sk8ai2jkf-user-environment.drv'...
created 2252 symlinks in user environment

$ plover
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, xcb.

fish: “plover” terminated by signal SIGABRT (Abort)

delroth commented 3 years ago

@Xe already tracked in #98069, and I think it's more similar to #98067 than this particular issue.

nixos-discourse commented 1 year ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/streamdeck-ui-on-unstable-getting-qt-error/27996/1

NixOS / nixpkgs

Tracking issue for wrapQtAppsHook #65399

Pull requests

65366

65372