search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.09k stars 14.07k forks source link

Build failure: QGIS #248239

Closed timlinux closed 1 year ago

timlinux commented 1 year ago

🏰Background

The QGIS packages by default are out of date from what is available upstream at https://qgis.org

My name is Tim (@timlinux) and I am a member of the QGIS community.

This issue tries to synthesize all of the learnings and issues I have relating to building the latest derivation on Nixos.

🗓️ Prior Work: The QGIS package (and other important dependant packages like GDAL etc.) are maintained by the Nixos Geospatial Team as per this announcement - my changes in my fork are just slight tweaks to their work to try to move us closer to having up to date packages on Nixos again.

🏃‍♂️My Workflow

🧑‍💻 I am working on a fork of the nixpkgs repo 🌳 Branch: nixos-unstable 📦️Package Path: pkgs/applications/gis/qgis 👷Building: In the top level nixpkgsgit repo folder by doing: nix build .#qgis 🏗️ Changes: The recipe contains two derivations (qgis and qgis.ltr). Since QGIS 3.28 the patching process has diverged a little, so I tweaked the patches appropriate to these versions and their dependencies.

🗃️ Files provided:

Entry Points

Patches

Derivations

⛔️ Build Errors

When running nix build .#qgis with my updated patches, we get the following error:

error: builder for '/nix/store/g6swjwrfhz2vyc3bf9kkc76bkws09iz4-qgis-unwrapped-3.32.1.drv' failed with exit code 1;
       last 10 log lines:
       > [9821/9833] Generating ui_DlgAddGeometryColumn.py
       > [9822/9833] Generating ui_DlgCreateIndex.py
       > [9823/9833] Generating ui_qgsplugininstallerpluginerrorbase.py
       > [9824/9833] Generating ui_qgsplugininstallerfetchingbase.py
       > [9825/9833] Generating ui_DlgCreateTable.py
       > [9826/9833] Generating ui_DlgTableProperties.py
       > [9827/9833] Generating ui_DlgVersioning.py
       > [9828/9833] Generating ui_DlgImportVector.py
       > [9829/9833] Generating ui_console_settings.py
       > ninja: build stopped: subcommand failed.
       For full logs, run 'nix log /nix/store/g6swjwrfhz2vyc3bf9kkc76bkws09iz4-qgis-unwrapped-3.32.1.drv'.
error: 1 dependencies of derivation '/nix/store/qc48nv3k3rjq4rsn2q62jg11cvff6j3k-qgis-3.32.1.drv' failed to build

🥼Diagnosis

I have spent a lot of time trying to figure out what is causing the issue. From this poking around, I understand the cause, as explained below, what needs to be done to fix it, but not how to fix it:

🧨 The cause

CMakeLists.txt

The cause of the issue is this line in the python CMakeLists.txt here: https://github.com/qgis/QGIS/blob/master/python/CMakeLists.txt#L367

COMMAND ${_python} "${APIS_SRC_DIR}/generate_console_pap.py" -platform offscreen "${QGIS_PYTHON_PAP_FILE}" "${APIS_SRC_DIR}" "${APIS_OUT_DIR}"

generate_console_pap.py

👩‍🏫 Explanation:

👩‍⚖️ Fix Attempts

With reverting to not using the -platform flag at all (as per LTR)

 [9808/9833] Generating pap file for console auto-completion
FAILED: python/qsci_apis/PyQGIS.pap /build/source/build/python/qsci_apis/PyQGIS.pap 
cd /build/source/build && /nix/store/2c7sgx69p6mmp76cvmi5j6c72dj76jj8-python3-3.10.12/bin/python3.10 /build/source/python/qsci_apis/generate_console_pap.py /bu>
qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in ""
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
Available platform plugins are: wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx.
[9809/9833] Generating ui_qgsplugininstallerfetchingbase.py
[9810/9833] Generating ui_DlgSqlWindow.py
[9811/9833] Generating ui_qgsplugininstallerpluginerrorbase.py
[9812/9833] Generating ui_console_compile_apis.py
[9813/9833] Generating ui_DlgAddGeometryColumn.py

When attempting to use one of the listed Qt platform plugins

-- Check for working CXX compiler: /nix/store/ygi2qf01dyx5015lqlqshrdh9gqijkxa-gcc-wrapper-12.3.0/bin/g++ - skipped
-- Detecting CXX compile features
...skipping...
Failed to create wl_display (No such file or directory)
Fontconfig error: Cannot load default config file: No such file: (null)
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
qt.qpa.plugin: Could not load the Qt platform plugin "wayland" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
Available platform plugins are: wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx.
[9809/9833] Generating ui_qgsplugindependenciesdialogbase.py
[9810/9833] Generating ui_DlgQueryBuilder.py
[9811/9833] Generating ui_DlgCancelTaskQuery.py
[9812/9833] Generating ui_DlgExportVector.py

It should be noted that my Nixos desktop is running on X11

Attempting to switch over to Gnome on Wayland with -platform wayland fails

Curious if compiling under wayland would make any difference, I changed my Nixos config to run under wayland. It still fails:

[9808/9833] Generating pap file for console auto-completion
FAILED: python/qsci_apis/PyQGIS.pap /build/source/build/python/qsci_apis/PyQGIS.pap 
cd /build/source/build && /nix/store/2c7sgx69p6mmp76cvmi5j6c72dj76jj8-python3-3.10.12/bin/python3.10 /build/source/python/qsci_apis/generate_console_pap.py -platform wayland /build/source/build/python/qsci_apis/P>
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
Failed to create wl_display (No such file or directory)
Fontconfig error: Cannot load default config file: No such file: (null)
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
qt.qpa.plugin: Could not load the Qt platform plugin "wayland" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
Available platform plugins are: wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx.
[9809/9833] Generating ui_DlgExportVector.py
[9810/9833] Generating ui_qgsplugininstallerfetchingbase.py
[9811/9833] Generating ui_DlgCancelTaskQuery.py
[9812/9833] Generating ui_DlgDbError.py
[9813/9833] Generating ui_console_settings.py
[9814/9833] Generating ui_qgsplugininstallerinstallingbase

Existing attempts to fix

Fixing the issue

I should have mentioned at the start of this ticket, I am a new Nixos user, so apologies for any misused terms etc. That status also means I don't have insight into how to configure the build process to ensure that the offscreen Qt QApplication plugin is available to QGIS, but I am pretty sure if this can be resolved (either e.g. as a flag to the build process or as an upstream fix to the Qt5 packaging), QGIS builds will work again.

Once we resolve this, I will make a PR and try to get QGIS building an up to date version again.

Notify maintainers

@imincik @sikmir @nh2 @willcohen

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

-
   nix-shell -p nix-info --run "nix-info -m"  
    - system: `"x86_64-linux"`
    - host os: `Linux 6.1.43, NixOS, 23.11 (Tapir), 23.11pre512200.41c760571839`
    - multi-user?: `yes`
    - sandbox: `yes`
    - version: `nix-env (Nix) 2.15.1`
    - channels(timlinux): `"nixos-unstable"`
    - channels(root): `"nixos"`
    - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
timlinux commented 1 year ago

Some further analysis:

I went spelunking in /nix/store a little:

ls | grep qtbase-5
7m1px7m2zvidm7w3s93qjgvpkvdzwp8c-qtbase-5.15.9-dev
9j0acz9qqp1lygwif5jncpz8hsyfmylw-qtbase-5.15.9-bin
b085hafh1ph7alj1nnbzmz29znni4v5w-qtbase-5.15.9
d1wlnn97g744ypyi57i10hd9y2v7r2x4-qtbase-5.15.9.drv
dg22lkxp1jfcyw1i971zlydi87w837ad-qtbase-5.15.9-dev
dqc9m05g5z0jkj8f3yq168qcd5x772z1-qtbase-5.15.9.drv
fbyd6iqs912slzw47ng5xm3k5hg0v56g-qtbase-5.15.9.drv
kqghllpfj5kcfyns1aldl7lmcszvz8nz-qtbase-5.15.9-bin
ph7x48dvr0mkv5dvjmkip2rswsv8yq5r-qtbase-5.15.9.drv
w0rqvc39zhlv1b633zim1rwva0hbij85-qtbase-5.15.9

We can see that in the bin folders, if we dig down, we can find the platform plugins folder.

ls 9j0acz9qqp1lygwif5jncpz8hsyfmylw-qtbase-5.15.9-bin/lib/qt-5.15.9/plugins/platforms
libqeglfs.so  libqlinuxfb.so  libqminimal.so  libqminimalegl.so  libqoffscreen.so  libqvnc.so  libqxcb.so

📝 In the above listing we can see the libqoffscreen.so that is the heart of our troubles:

image

So next i will try to match which derivation is being used by our QGIS build and understand what platform plugins are listed there.

timlinux commented 1 year ago

Another nugget (which you will see is commented out in unwrapped.nix) is to add

QT_QPA_PLATFORM_PLUGIN_PATH="${qt5.qtbase.bin}/lib/qt-${qt5.qtbase.versi on}/plugins/platforms"; To the environment, though I am still unsure how to do that,

nh2 commented 1 year ago

@timlinux This is how you can add environment variables for the runtime:

https://github.com/NixOS/nixpkgs/blob/7eefa92432ad9234b349f943e02285643ca70445/pkgs/applications/graphics/pixinsight/default.nix#L111

imincik commented 1 year ago

Hi @timlinux , sorry for not replying sooner, but I just returned from my vacation and I am just catching up on notifications. I am very excited to see you interested in Nix.

I was trying to update QGIS in https://github.com/NixOS/nixpkgs/pull/224805 ,but as you saw, I failed there. Yes, it looks like that QGIS can't find offscreen for some reason. I'll try to set QT_QPA_PLATFORM_PLUGIN_PATH and will see if it helps.

Do you have any update since your last comment ?

BTW, you are very welcome to join Nix Geospatial Team.

timlinux commented 1 year ago

Hi @timlinux , sorry for not replying sooner, but I just returned from my vacation and I am just catching up on notifications. I am very excited to see you interested in Nix.

I was trying to update QGIS in #224805 ,but as you saw, I failed there. Yes, it looks like that QGIS can't find offscreen for some reason. I'll try to set QT_QPA_PLATFORM_PLUGIN_PATH and will see if it helps.

Do you have any update since your last comment ?

BTW, you are very welcome to join Nix Geospatial Team.

Hi @imincik

Thanks very much. So I will share my updates soon, but I have still been hacking away trying to make things work. I am looking at three variants:

I don't know if there is any broader appeal of the without python variant, but I would probable suggest merging in my WITH_QSCIAPI=False changed version for now since it is a better outcome than having no latest release available at all. We can just annotate this caveat in a README in the interim.

I'll put another update here soon with the link to the actual changes and eventually offer a PR.

For the longer term fix, I need the help of someone who understands nix better because, from what I can make out, the python/CMakeLists.txt file needs to call this generate_console_pap.py with the proper Qt platform plugins available to it.

367 COMMAND ${_python} "${APIS_SRC_DIR}/generate_console_pap.py" -platform offscreen "${QGIS_PYTHON_PAP_FILE}" "${APIS_SRC_DIR}" "${APIS_OUT_DIR}"

timlinux commented 1 year ago

image

timlinux commented 1 year ago

@imincik Ok PR made (see link above) - would love to get your feedback.

Still no GRASS and SAGA support which I would like to figure out still....