nix-community / poetry2nix

Convert poetry projects to nix automagically [maintainer=@adisbladis,@cpcloud]
MIT License
837 stars 435 forks source link

PySide6 broken Overrides #1791

Open zierf opened 1 week ago

zierf commented 1 week ago

Describe the issue

I tried to create a devShell and derivation for the TagStudio project using poetry2nix. Unfortunately, the overrides provided as standard no longer seem to work.

Unfortunately, the overrides provided as standard no longer seem to work and I had to resort to this trick, which replaces the entire package with with the one from nixpkgs. But this would require both versions to remain consistent with each other in the long term.

Not using any overrides at all will result in the following error.

Build Error with just default overrides ```SH $> nix run auto-patchelf: 35 dependencies could not be satisfied warn: auto-patchelf ignoring missing libQt6WebEngineWidgets.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/designer/libqwebengineview.so warn: auto-patchelf ignoring missing libQt6WebEngineCore.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/designer/libqwebengineview.so warn: auto-patchelf ignoring missing libQt6WebChannel.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/designer/libqwebengineview.so warn: auto-patchelf ignoring missing libQt6Positioning.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/designer/libqwebengineview.so warn: auto-patchelf ignoring missing libQt6EglFsKmsGbmSupport.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/egldeviceintegrations/libqeglfs-kms-integration.so error: auto-patchelf could not satisfy dependency libgbm.so.1 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/egldeviceintegrations/libqeglfs-kms-integration.so warn: auto-patchelf ignoring missing libQt6Pdf.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/imageformats/libqpdf.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboard.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/platforminputcontexts/libqtvirtualkeyboardplugin.so warn: auto-patchelf ignoring missing libQt6Quick3DUtils.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/qmltooling/libqmldbg_quick3dprofiler.so warn: auto-patchelf ignoring missing libmimerapi.so wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/sqldrivers/libqsqlmimer.so warn: auto-patchelf ignoring missing libmysqlclient.so.21 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/plugins/sqldrivers/libqsqlmysql.so warn: auto-patchelf ignoring missing libQt6ShaderTools.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/Qt5Compat/GraphicalEffects/private/libqtgraphicaleffectsprivateplugin.so warn: auto-patchelf ignoring missing libQt6StateMachineQml.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQml/StateMachine/libqtqmlstatemachineplugin.so warn: auto-patchelf ignoring missing libQt6StateMachine.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQml/StateMachine/libqtqmlstatemachineplugin.so warn: auto-patchelf ignoring missing libQt6PdfQuick.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Pdf/libpdfquickplugin.so warn: auto-patchelf ignoring missing libQt6Pdf.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Pdf/libpdfquickplugin.so warn: auto-patchelf ignoring missing libQt63DQuickScene2D.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Scene2D/libqtquickscene2dplugin.so warn: auto-patchelf ignoring missing libQt63DQuick.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Scene2D/libqtquickscene2dplugin.so warn: auto-patchelf ignoring missing libQt63DRender.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Scene2D/libqtquickscene2dplugin.so warn: auto-patchelf ignoring missing libQt63DCore.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Scene2D/libqtquickscene2dplugin.so warn: auto-patchelf ignoring missing libQt63DInput.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Scene3D/libqtquickscene3dplugin.so warn: auto-patchelf ignoring missing libQt63DLogic.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Scene3D/libqtquickscene3dplugin.so warn: auto-patchelf ignoring missing libQt63DAnimation.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Scene3D/libqtquickscene3dplugin.so warn: auto-patchelf ignoring missing libQt63DRender.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Scene3D/libqtquickscene3dplugin.so warn: auto-patchelf ignoring missing libQt63DCore.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/Scene3D/libqtquickscene3dplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboard.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/libqtvkbplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboard.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/Components/libqtvkbcomponentsplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboardSettings.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/Components/libqtvkbcomponentsplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboardSettings.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/Settings/libqtvkbsettingsplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboard.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/Styles/libqtvkbstylesplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboard.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/Plugins/Hangul/libqtvkbhangulplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboard.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/Plugins/OpenWNN/libqtvkbopenwnnplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboard.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/Plugins/Pinyin/libqtvkbpinyinplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboard.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/Plugins/TCIme/libqtvkbtcimeplugin.so warn: auto-patchelf ignoring missing libQt6VirtualKeyboard.so.6 wanted by /nix/store/d80dqhi4g3wsb56pdc9991zbhymzl01v-python3.12-pyside6-essentials-6.7.2/lib/python3.12/site-packages/PySide6/Qt/qml/QtQuick/VirtualKeyboard/Plugins/Thai/libqtvkbthaiplugin.so auto-patchelf failed to find all the required dependencies. Add the missing dependencies to --libs or use `--ignore-missing="foo.so.1 bar.so etc.so"`. ```

However, these can at least be circumvented with libgbm.so.1 in autoPatchelfIgnoreMissingDeps and three other new propagatedBuildInputs for pyside6-essentials.

pkgs.qt6.qtquick3d
pkgs.qt6.qtvirtualkeyboard
pkgs.qt6.qtwebengine

The build itself then runs through, but a runtime error is thrown when the program is executed.

Example Runtime Error after extending overrides ```SH $> nix run Traceback (most recent call last): File "/nix/store/lkih98vq66kb3cfw3j961bln7wd30843-python3.12-tagstudio-9.3.2/bin/..tagstudio-wrapped-wrapped", line 6, in from tagstudio.tag_studio import main File "/nix/store/lkih98vq66kb3cfw3j961bln7wd30843-python3.12-tagstudio-9.3.2/lib/python3.12/site-packages/tagstudio/tag_studio.py", line 8, in from src.cli.ts_cli import CliDriver # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/lkih98vq66kb3cfw3j961bln7wd30843-python3.12-tagstudio-9.3.2/lib/python3.12/site-packages/tagstudio/src/cli/ts_cli.py", line 32, in from src.qt.helpers.file_opener import open_file File "/nix/store/lkih98vq66kb3cfw3j961bln7wd30843-python3.12-tagstudio-9.3.2/lib/python3.12/site-packages/tagstudio/src/qt/helpers/file_opener.py", line 13, in from PySide6.QtWidgets import QLabel ModuleNotFoundError: No module named 'PySide6.QtWidgets' ```

Interestingly, it works with poetry run, but I suspect that this is due to the internally used venv.

I have seen the section regarding ModuleNotFoundError in the Edge cases, but unfortunately this only refers to the build (and was used as described above for the pyside6-essentials overlay).


Since it turned out to be a more difficult undertaking than expected, I created a minimal poetry2nix_pyside6 project, which contains less code that is not relevant to the problem and therefore should build and start faster. Maybe it can also be used as a template for a PySide6 project in future. (The main branch uses the trick mentioned above as a workaround, the _overridespyside6 branch is one of my attempts to provide suitable overrides)

Here are the logs specific to the example project.

$> poetry run poetry2nix-example

Loaded External Module: <class 'PySide6.QtCore.QPoint'>

Message from subfolder.
$> nix run

Traceback (most recent call last):
  File "/nix/store/yh68dpma2422rqfq9wr9g3fbjna5cwaz-python3.12-poetry2nix-example-0.1.0/bin/..poetry2nix-example-wrapped-wrapped", line 6, in <module>
    from poetry2nix_example.example import main
  File "/nix/store/yh68dpma2422rqfq9wr9g3fbjna5cwaz-python3.12-poetry2nix-example-0.1.0/lib/python3.12/site-packages/poetry2nix_example/example.py", line 1, in <module>
    from PySide6.QtCore import QPoint
ModuleNotFoundError: No module named 'PySide6.QtCore'

For a Flake related to the TagStudio project, you can refer to zierf/TagStudio/poetry2nix. This is a fork of TagStudio, which should help to create a working derivation with poetry2nix later on (a pull request for the general use of Poetry is already in progress).

I have also already listed some additional libraries in it, such as qt6.qtmultimedia/xorg.libXrandr/libva for video playback and hardware acceleration and openssl, which were still partly missing despite using pkgs.python312.pkgs.pyside6 as a temporary workaround.

I really hope that someone can help solve this problem and correct the default overrides for PySide6 in poetry2nix. Maybe my failed attempt can help solve it.


Additional context

Files for minimal example at zierf/poetry2nix_pyside6/overrides_pyside6.

flake.nix ```nix { description = "Minimal Example for Nix2Poetry"; inputs = { nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; flake-utils.url = "github:numtide/flake-utils"; poetry2nix = { url = "github:nix-community/poetry2nix"; inputs.nixpkgs.follows = "nixpkgs"; }; }; outputs = { self, nixpkgs, flake-utils, poetry2nix, ... } @inputs: flake-utils.lib.eachDefaultSystem (system: let pkgs = nixpkgs.legacyPackages.${system}.extend poetry2nix.overlays.default; inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication mkPoetryEnv defaultPoetryOverrides; myApp = mkPoetryApplication { projectDir = self; exeName = "poetry2nix-example"; python = pkgs.python312; preferWheels = true; # extend official overrides # https://github.com/nix-community/poetry2nix/blob/7619e43c2b48c29e24b88a415256f09df96ec276/overrides/default.nix#L2743-L2805 overrides = defaultPoetryOverrides.extend (final: prev: { # working Overrides for PySide6 via nixpkgs # https://github.com/nix-community/poetry2nix/issues/1191#issuecomment-1707590287 #pyside6 = final.pkgs.python312.pkgs.pyside6; #shiboken6 = final.pkgs.python3.pkgs.shiboken6; pyside6 = prev.pyside6.overridePythonAttrs (old: pkgs.lib.optionalAttrs pkgs.stdenv.isLinux { dontWrapQtApps = true; # also try PatchElf search paths for main package preFixup = '' addAutoPatchelfSearchPath $out/${final.python.sitePackages}/PySide6 addAutoPatchelfSearchPath ${final.shiboken6}/${final.python.sitePackages}/shiboken6 ''; postInstall = '' rm -r $out/${final.python.sitePackages}/PySide6/__pycache__ ''; propagatedBuildInputs = old.propagatedBuildInputs or [ ] ++ [ pkgs.qt6.qtmultimedia # dependencies from nixpkgs pyside6 # https://github.com/NixOS/nixpkgs/blob/c40ce90d28e607d3b967963a0240c43d1210dbc5/pkgs/development/python-modules/pyside6/default.nix pkgs.qt6.qtbase # optional pkgs.qt6.qt3d pkgs.qt6.qtcharts pkgs.qt6.qtconnectivity pkgs.qt6.qtdatavis3d pkgs.qt6.qtdeclarative pkgs.qt6.qthttpserver pkgs.qt6.qtmultimedia pkgs.qt6.qtnetworkauth pkgs.qt6.qtquick3d pkgs.qt6.qtremoteobjects pkgs.qt6.qtscxml pkgs.qt6.qtsensors pkgs.qt6.qtspeech pkgs.qt6.qtsvg pkgs.qt6.qtwebchannel pkgs.qt6.qtwebsockets pkgs.qt6.qtpositioning pkgs.qt6.qtlocation pkgs.qt6.qtshadertools pkgs.qt6.qtserialport pkgs.qt6.qtserialbus pkgs.qt6.qtgraphs pkgs.qt6.qttools ]; }); pyside6-essentials = prev.pyside6-essentials.overridePythonAttrs (old: pkgs.lib.optionalAttrs pkgs.stdenv.isLinux { # prevent error: 'Error: wrapQtAppsHook is not used, and dontWrapQtApps is not set.' dontWrapQtApps = true; # added libgbm.so.1 to satsify missing library for auto-patchelf autoPatchelfIgnoreMissingDeps = [ "libmysqlclient.so.21" "libmimerapi.so" "libQt6*" "libgbm.so.1" ]; preFixup = '' addAutoPatchelfSearchPath $out/${final.python.sitePackages}/PySide6 addAutoPatchelfSearchPath ${final.shiboken6}/${final.python.sitePackages}/shiboken6 ''; postInstall = '' rm -r $out/${final.python.sitePackages}/PySide6/__pycache__ ''; propagatedBuildInputs = old.propagatedBuildInputs or [ ] ++ [ # PySide6 in nixpkgs needs these for video playback pkgs.qt6.qtmultimedia pkgs.xorg.libXrandr # enable hardware acceleration pkgs.libva pkgs.openssl # satisfy some missing libraries for auto-patchelf patching PySide6-Essentials pkgs.qt6.qtquick3d pkgs.qt6.qtvirtualkeyboard pkgs.qt6.qtwebengine # other packages from default override pkgs.libxkbcommon pkgs.gtk3 pkgs.speechd pkgs.gst pkgs.gst_all_1.gst-plugins-base pkgs.gst_all_1.gstreamer pkgs.postgresql.lib pkgs.unixODBC pkgs.pcsclite pkgs.xorg.libxcb pkgs.xorg.xcbutil pkgs.xorg.xcbutilcursor pkgs.xorg.xcbutilerrors pkgs.xorg.xcbutilimage pkgs.xorg.xcbutilkeysyms pkgs.xorg.xcbutilrenderutil pkgs.xorg.xcbutilwm pkgs.libdrm pkgs.pulseaudio final.shiboken6 ]; }); pyside6-addons = prev.pyside6-addons.overridePythonAttrs (old: pkgs.lib.optionalAttrs pkgs.stdenv.isLinux { dontWrapQtApps = true; autoPatchelfIgnoreMissingDeps = [ "libmysqlclient.so.21" "libmimerapi.so" "libQt6Quick3DSpatialAudio.so.6" "libQt6Quick3DHelpersImpl.so.6" ]; preFixup = '' addAutoPatchelfSearchPath ${final.shiboken6}/${final.python.sitePackages}/shiboken6 addAutoPatchelfSearchPath ${final.pyside6-essentials}/${final.python.sitePackages}/PySide6 ''; propagatedBuildInputs = old.propagatedBuildInputs or [ ] ++ [ pkgs.nss pkgs.xorg.libXtst pkgs.alsa-lib pkgs.xorg.libxshmfence pkgs.xorg.libxkbfile ]; postInstall = '' rm -r $out/${final.python.sitePackages}/PySide6/__pycache__ ''; }); }); pythonRelaxDeps = [ ]; buildInputs = (with pkgs; [ qt6.qtbase qt6.qtmultimedia ]); nativeBuildInputs = (with pkgs; [ makeWrapper qt6.wrapQtAppsHook ]); propogatedBuildInputs = (with pkgs; [ ]); }; in { # $> nix run packages = { myApp = myApp; default = myApp; }; # Development Shell including `poetry`. # $> nix develop # # Use this shell for developing the application and # making changes to `pyproject.toml` and `poetry.lock` files. # # $> poetry install => install packages stated by petry.lock file # $> poetry lock => update lock file after changing dependencies # $> python -m poetry2nix_example.example => launch application as Python module # $> poetry run poetry2nix-example => execute the application via Poetry devShells.default = pkgs.mkShell rec { #inputsFrom = [ self.apps.${system}.default ]; inputsFrom = [ self.packages.${system}.myApp ]; packages = with pkgs; [ poetry ]; buildInputs = (with pkgs; [ stdenv.cc.cc.lib dbus fontconfig freetype glib libGL libkrb5 libpulseaudio libva libxkbcommon openssl # video playback xorg.libXrandr ]); LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath (packages ++ buildInputs); }; # Shell for poetry. # # Needed to create first lock file. # $> nix develop .#poetry # $> poetry install # # Use this shell for changes to pyproject.toml and poetry.lock. devShells.poetry = pkgs.mkShell { packages = [ pkgs.poetry ]; }; } ); } ```
pyproject.toml ```toml [tool.ruff] exclude = ["main_window.py", "home_ui.py", "resources.py", "resources_rc.py"] [tool.mypy] strict_optional = false disable_error_code = ["union-attr", "annotation-unchecked", "import-untyped"] explicit_package_bases = true warn_unused_ignores = true exclude = ['tests'] [tool.poetry] name = "poetry2nix_example" version = "0.1.0" description = "Minimal Example for Nix2Poetry" authors = ["zierf <9168602+zierf@users.noreply.github.com>"] license = "MIT" packages = [ { include = "poetry2nix_example", from = "." }, ] [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" [tool.poetry.dependencies] python = "~3.12" PySide6 = "6.7.2" # PySide6 includes PySide6_Essentials and PySide6_Addons #PySide6_Addons= "6.7.2" #PySide6_Essentials = "6.7.2" [tool.poetry.group.dev] optional = true [tool.poetry.group.dev.dependencies] [tool.poetry.scripts] poetry2nix-example = 'poetry2nix_example.example:main' ```
poetry.lock ```toml # This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand. [[package]] name = "pyside6" version = "6.7.2" description = "Python bindings for the Qt cross-platform application and UI framework" optional = false python-versions = "<3.13,>=3.9" files = [ {file = "PySide6-6.7.2-cp39-abi3-macosx_11_0_universal2.whl", hash = "sha256:602debef9ec159b0db48f83b38a0e43e2dad3961f7d99f708d98620f04e9112b"}, {file = "PySide6-6.7.2-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:15e7696a09072ee977f6e6179ab1e48184953df8417bcaa83cfadf0b79747242"}, {file = "PySide6-6.7.2-cp39-abi3-manylinux_2_31_aarch64.whl", hash = "sha256:6e0acb471535de303f56e3077aa86f53496b4de659b99ecce80520bcee508a63"}, {file = "PySide6-6.7.2-cp39-abi3-win_amd64.whl", hash = "sha256:f73ae0de77d67f51ca3ce8207b12d3a5fa0107d3d5b6e4aeb3b53ee842b0927a"}, ] [package.dependencies] PySide6-Addons = "6.7.2" PySide6-Essentials = "6.7.2" shiboken6 = "6.7.2" [[package]] name = "pyside6-addons" version = "6.7.2" description = "Python bindings for the Qt cross-platform application and UI framework (Addons)" optional = false python-versions = "<3.13,>=3.9" files = [ {file = "PySide6_Addons-6.7.2-cp39-abi3-macosx_11_0_universal2.whl", hash = "sha256:90b995efce61058d995c603ea480a9a3054fe8206739dcbc273fc3b53d40650f"}, {file = "PySide6_Addons-6.7.2-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:94b9bf6a2a4a7ac671e1776633e50d51326c86f4184f1c6e556f4dd5498fd52a"}, {file = "PySide6_Addons-6.7.2-cp39-abi3-manylinux_2_31_aarch64.whl", hash = "sha256:22979b1aa09d9cf1d7a86c8a9aa0cb4791d6bd1cc94f96c5b6780c5ef8a9e34e"}, {file = "PySide6_Addons-6.7.2-cp39-abi3-win_amd64.whl", hash = "sha256:ebf549eb25998665d8e4ec24014fbbd37bebc5ecdcb050b34db1e1c03e1bf81d"}, ] [package.dependencies] PySide6-Essentials = "6.7.2" shiboken6 = "6.7.2" [[package]] name = "pyside6-essentials" version = "6.7.2" description = "Python bindings for the Qt cross-platform application and UI framework (Essentials)" optional = false python-versions = "<3.13,>=3.9" files = [ {file = "PySide6_Essentials-6.7.2-cp39-abi3-macosx_11_0_universal2.whl", hash = "sha256:4d13666e796ec140ecfb432c4f3d7baef6dfafc11929985a83b22c0025532fb7"}, {file = "PySide6_Essentials-6.7.2-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:a1a4c09f1e916b9cfe53151fe4a503a6acb1f6621ba28204d1bfe636f80d6780"}, {file = "PySide6_Essentials-6.7.2-cp39-abi3-manylinux_2_31_aarch64.whl", hash = "sha256:9135513e1c4c6e2fbb1e4f9afcb3d42e54708b0d9ed870cb3213ea4874cafa1e"}, {file = "PySide6_Essentials-6.7.2-cp39-abi3-win_amd64.whl", hash = "sha256:0111d5fa8cf826de3ca9d82fed54726cce116d57f454f88a6467578652032d69"}, ] [package.dependencies] shiboken6 = "6.7.2" [[package]] name = "shiboken6" version = "6.7.2" description = "Python/C++ bindings helper module" optional = false python-versions = "<3.13,>=3.9" files = [ {file = "shiboken6-6.7.2-cp39-abi3-macosx_11_0_universal2.whl", hash = "sha256:50c33ac6317b673a1eb97a9abaafccb162c4ba0c9ca658a8e449c49a8aadc379"}, {file = "shiboken6-6.7.2-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:70e80737b27cd5d83504b373013b55e70462bd4a27217d919ff9a83958731990"}, {file = "shiboken6-6.7.2-cp39-abi3-manylinux_2_31_aarch64.whl", hash = "sha256:98bedf9a15f1d8ba1af3e4d1e7527f7946ce36da541e08074fd9dc9ab5ff1adf"}, {file = "shiboken6-6.7.2-cp39-abi3-win_amd64.whl", hash = "sha256:9024e6afb2af1568ebfc8a5d07e4ff6c8829f40923eeb28901f535463e2b6b65"}, ] [metadata] lock-version = "2.0" python-versions = "~3.12" content-hash = "4ef9d6cee1e36112df3b6339a469eb6d6954010617d3c2daf13e404b73ef7ab6" ```
zierf commented 1 week ago

I was wondering why the overrides of PySide6 v6.6.0 might have worked back then and don't work anymore today.

So I created a branch legacy_pyside6-6-0 with the old version for poetry2nix (Commit #2d29b1 from Pull-Request #1356) and nixpkgs from that day. It looks like the package could only be built and it throws the same errors at runtime.

$> nix run

Traceback (most recent call last):
  File "/nix/store/53vxf97jval3xcfsx1lgfhcqcgk5df6x-python3.10-poetry2nix-example-0.1.0/bin/..poetry2nix-example-wrapped-wrapped", line 6, in <module>
    from poetry2nix_example.example import main
  File "/nix/store/53vxf97jval3xcfsx1lgfhcqcgk5df6x-python3.10-poetry2nix-example-0.1.0/lib/python3.10/site-packages/poetry2nix_example/example.py", line 1, in <module>
    from PySide6.QtCore import QPoint
ModuleNotFoundError: No module named 'PySide6.QtCore'
$> python -m poetry2nix_example.example

Traceback (most recent call last):
  File "/nix/store/s6fgyqbk8vn1014daznm5kqx90xdn86x-python3-3.10.13/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/nix/store/s6fgyqbk8vn1014daznm5kqx90xdn86x-python3-3.10.13/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/zierf/workspaces/Python/PySide6Minimal/poetry2nix_example/example.py", line 1, in <module>
    from PySide6.QtCore import QPoint
ModuleNotFoundError: No module named 'PySide6.QtCore'

The tests only covered the successful build.

@considerate is there any chance you could take another look at this and expand on the overrides from back then?

cromachina commented 1 week ago

I tried out the override in your example project, but when it comes down to actually running a non-trival app, like say something that includes more of the core components

from PySide6.QtCore import *
from PySide6.QtGui import *
from PySide6.QtWidgets import *

and then subsequently running the app in the flake dev shell nix develop:

[cro@titan:~/aux/Projects/crowpainter]$ poetry run crowpainter
qt.qpa.plugin: From 6.5.0, xcb-cursor0 or libxcb-cursor0 is needed to load the Qt xcb platform plugin.
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/cro/.cache/pypoetry/virtualenvs/crowpainter-iuzixrfN-py3.12/lib/python3.12/site-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: minimal, linuxfb, wayland-egl, eglfs, xcb, wayland, minimalegl, offscreen, vnc, vkkhrdisplay.

Aborted (core dumped)

I get errors that seem to be related to wrapping executables with wrapQtAppsHook (or lack thereof).

cromachina commented 1 week ago

After running my project with QT_DEBUG_PLUGINS=1 and seeing the exact libraries it wasn't able to load, it turns out that the flake needed a few more dependencies, mainly the xcb util stuff. There are probably even more dependencies that need to be added to cover everything Qt might possibly load at runtime. Unfortunately looking at pyside6's nix derivation doesn't give much of a clue, at least for me.

          dependencies = (with pkgs; [
            zlib
            dbus
            fontconfig
            freetype
            glib
            libGL
            libkrb5
            libpulseaudio
            libva
            libxkbcommon
            openssl
            qt6.full
            stdenv.cc.cc.lib
            wayland
            xorg.libX11
            xorg.libxcb
            xorg.libXi
            xorg.libXrandr
            xorg.xcbutilwm
            xorg.xcbutilimage
            xorg.xcbutilkeysyms
            xorg.xcbutilrenderutil
            xorg.xcbutilerrors
            xcb-util-cursor
          ]);
zierf commented 1 week ago

Hey, thanks for trying. The big problem is that Poetry most likely always uses a venv under the hood during development. If you add a few libraries that are missing in the example for a working QT app, you can almost always get it to run with poetry run.

Make it work with poetry run Just add your additional defined dependencies to the `LD_LIBRARY_PATH` in the `devShells.default` and `poetry run` should work for you. If I defined the attribute `dependencies` under `myApp`: ```Nix LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath ( packages ++ buildInputs ++ self.packages.${system}.myApp.dependencies ); ``` And then run the project with _Poetry_: ```SH $> poetry run poetry2nix-example Loaded External Module: Message from subfolder. ```

The main problem always exists after building the derivation, after which the QT* modules of the Python modules themselves can no longer be found at runtime.

Same project, just with nix run instead of poetry run.

$> nix run

Traceback (most recent call last):
  File "/nix/store/h9vcjm1rh73xil6ch9q5n0jn0q1xcjia-python3.12-poetry2nix-example-0.1.0/bin/..poetry2nix-example-wrapped-wrapped", line 6, in <module>
    from poetry2nix_example.example import main
  File "/nix/store/h9vcjm1rh73xil6ch9q5n0jn0q1xcjia-python3.12-poetry2nix-example-0.1.0/lib/python3.12/site-packages/poetry2nix_example/example.py", line 1, in <module>
    from PySide6.QtCore import *
ModuleNotFoundError: No module named 'PySide6.QtCore'

Some system libraries that may be missing at the end will probably be a combination of what we have already listed in the Flake for Tagstudio and what the package python312Packages.pyside6 is using too. Unfortunately, I cannot get the PySide6 Python package to find its own modules at runtime.

After building the derivation, it simply forgets its correct reference. Everything is only usable in the devShell via the venv of Poetry. However, this means that the program cannot be made available to normal users.

cromachina commented 1 week ago

Shouldn't we technically be always running with in the scope of the venv anyway, because we want the python package isolation? Like calling nix run could do just that, enter the nix shell (or environment of the derivation) and call poetry run myapp, and any final derivation would also do that under the hood 🤔. It feels like a very nix wrapper kind of solution.

zierf commented 1 week ago

If I'm not mistaken, poetry2nix packages the various modules into their own derivations, similar to the python packages in nixpkgs. The venv are more of an easy way around this, which allows you to progress faster in development.

Python makes the already challenging Nix packaging hard enough, but also having a QT application in python makes even that much more difficult.

An application bundled with mkPoetryApplication should actually be able to run without an additional wrapper, but the QT modules just don't load correctly at runtime.

cromachina commented 1 week ago

Ah I see, that makes sense. Given the error you get from nix run, it's as if the environment doesn't even have pyside6 inside of it at all. Similarly, when I tried nix run on my own project, I have a custom build.py script for compiling Cython modules, and the build quickly fails because it cannot find distutils. I wonder if this failure during the build phase gives a hint at what might also be the issue with missing modules in your example project even after a successful build?

zierf commented 1 week ago

If you get past distutils during the build, you'll end up in basically the same situation. I've provided you with a pull request that provides distutils for your project.

The goal here is to not use the specially built python312.pkgs.pyside6 and to find an override where poetry2nix can fetch the Python package and patch it correctly. Otherwise, nixpkgs, Python version and PySide6 must always match, which isn't always possible.