qt6.qtbase: qmake looking for lrelease in wrong package

OPNA2608 commented 1 year ago

Describe the bug

When using Qt6's qmake to build a project that compiles its localisation with ~ this QMake code

TRANSLATIONS += lang_fr.ts
CONFIG += lrelease

compilation fails due to qmake emitting the wrong path to lrelease:

make[1]: Entering directory '/build/source/BambooTracker'
/nix/store/axbwbk9697fzj4akqajbh1sw653c72yl-qtbase-6.4.2-dev/bin/lrelease lang/bamboo_tracker_fr.ts -qm .qm/bamboo_tracker_fr.qm
/nix/store/qqa28hmysc23yy081d178jfd9a1yk8aw-bash-5.2-p15/bin/bash: line 1: /nix/store/axbwbk9697fzj4akqajbh1sw653c72yl-qtbase-6.4.2-dev/bin/lrelease: No such file or directory
make[1]: *** [Makefile:1358: .qm/bamboo_tracker_fr.qm] Error 127

The path it expects lrelease to be at is <qt6.qtbase.dev>/bin/lrelease, while it's actually at <qt6.qttools.dev>/bin/lrelease.

With Qt5, this works fine:

make[1]: Entering directory '/build/source/BambooTracker'
/nix/store/qyf983nr7l3mr8788486d1vv8w7y7qn5-qttools-5.15.8-dev/bin/lrelease lang/bamboo_tracker_fr.ts -qm .qm/bamboo_tracker_fr.qm
Updating '.qm/bamboo_tracker_fr.qm'...

Steps To Reproduce

Steps to reproduce the behavior:

Find a package that should build on both Qt5 and Qt6 via qmake (See Additional context for an example)
nix-build --no-out-link -E 'with import <nixpkgs> { }; libsForQt5.callPackage ./that-package.nix { }' works
nix-build --no-out-link -E 'with import <nixpkgs> { }; qt6Packages.callPackage ./that-package.nix { }' fails

Expected behavior

qmake knows where to find lrelease.

Screenshots

n/a

Additional context

bambootracker, adjusted to build with Qt5 and Qt6:

```nix { stdenv , lib , fetchFromGitHub , qmake , pkg-config , qttools , qtbase , qt5compat ? null , rtaudio , rtmidi , wrapQtAppsHook }: assert (lib.versionAtLeast qtbase.version "6.0") -> qt5compat != null; stdenv.mkDerivation rec { pname = "bambootracker"; version = "0.6.1"; src = fetchFromGitHub { owner = "BambooTracker"; repo = "BambooTracker"; rev = "v${version}"; fetchSubmodules = true; sha256 = "sha256-Ymi1tjJCgStF0Rtseelq/YuTtBs2PrbF898TlbjyYUw="; }; nativeBuildInputs = [ pkg-config qmake qttools wrapQtAppsHook ]; buildInputs = [ qtbase rtaudio rtmidi ] ++ lib.optionals (lib.versionAtLeast qtbase.version "6.0") [ qt5compat ]; qmakeFlags = [ "CONFIG+=system_rtaudio" "CONFIG+=system_rtmidi" ]; postConfigure = "make qmake_all"; # Manually wrapping on Darwin dontWrapQtApps = stdenv.hostPlatform.isDarwin; postInstall = lib.optionalString stdenv.hostPlatform.isDarwin '' mkdir -p $out/Applications mv $out/{bin,Applications}/BambooTracker.app ln -s $out/{Applications/BambooTracker.app/Contents/MacOS,bin}/BambooTracker wrapQtApp $out/Applications/BambooTracker.app/Contents/MacOS/BambooTracker ''; meta = with lib; { description = "A tracker for YM2608 (OPNA) which was used in NEC PC-8801/9801 series computers"; homepage = "https://bambootracker.github.io/BambooTracker/"; license = licenses.gpl2Plus; platforms = platforms.all; maintainers = with maintainers; [ OPNA2608 ]; mainProgram = "BambooTracker"; }; } ```

Notify maintainers

@milahu @NickCao

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.89-xanmod1, NixOS, 22.11 (Raccoon), 22.11.2050.cc4bb87f545`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.11.1`
 - channels(bt1cn): `"unstable"`
 - channels(root): `"nixos-22.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

milahu commented 1 year ago

/nix/store/8l5a48mrxfnfr1fnq4aa2zcyf183c942-qtbase-6.4.0-dev/mkspecs/features/lrelease.prf

lrelease.commands = $$QMAKE_LRELEASE ${QMAKE_FILE_IN} $$QMAKE_LRELEASE_FLAGS -qm ${QMAKE_FILE_OUT}

please grep your makefiles for QMAKE_LRELEASE ... where is it defined?

edit: probably not defined. by default, qmake uses binaries from its own dirname

/nix/store/8l5a48mrxfnfr1fnq4aa2zcyf183c942-qtbase-6.4.0-dev/bin/qmake
/nix/store/8l5a48mrxfnfr1fnq4aa2zcyf183c942-qtbase-6.4.0-dev/bin/lrelease

maybe try to set the path in your project file like

QMAKE_LRELEASE = /nix/store/...

potential fix

echo QMAKE_LRELEASE = ${qttools}/bin/lrelease >> $dev/mkspecs/features/lrelease.prf

OPNA2608 commented 1 year ago

QMAKE_LRELEASE is determined by QMake itself.

/nix/store/axbwbk9697fzj4akqajbh1sw653c72yl-qtbase-6.4.2-dev/mkspecs/features/lrelease.prf:10:qtPrepareTool(QMAKE_LRELEASE, lrelease) /nix/store/axbwbk9697fzj4akqajbh1sw653c72yl-qtbase-6.4.2-dev/mkspecs/features/qt_functions.prf:90:defineTest(qtPrepareTool) {

Qt5's qtPrepareTool is patched to deal with the binary fragmentation, is Qt6 perhaps missing that patching? pkgs/development/libraries/qt-5/5.15/qtbase.patch.d/0003-qtbase-mkspecs.patch

milahu commented 1 year ago

resolving the binareis with command -v looks like a quick hack, this breaks when a different lrelease is in $PATH

the only way to override lrelease should be via QMAKE_LRELEASE

problem is that qttools depends on qtbase, so we need a setupHook

... or we move ${qt6.qtbase.dev}/mkspecs/features/lrelease.prf to ${qt6.qttools.dev}/mkspecs/features/lrelease.prf with

-qtPrepareTool(QMAKE_LRELEASE, lrelease
+qtPrepareTool(QMAKE_LRELEASE, /nix/store/xxx-qttools-dev/bin/lrelease

→ no

/nix/store/gskwk8h8rirfq27hmmm4963aillnziv4-qtbase-6.4.2-dev/bin//nix/store/v9lkwz407hsqskxcraz6qafblb0vwwv2-qttools-6.4.2-dev/bin/lrelease i18n/qml_en.ts -qm .qm/qml_en.qm /nix/store/pj1hnyxhcsw1krmhnbb9rjvqssbzliw8-bash-5.2-p15/bin/bash: line 1: /nix/store/gskwk8h8rirfq27hmmm4963aillnziv4-qtbase-6.4.2-dev/bin//nix/store/v9lkwz407hsqskxcraz6qafblb0vwwv2-qttools-6.4.2-dev/bin/lrelease: No such file or directory

→ patch cmd = $$instloc/$$2 with something like cmd = $$system("realpath --relative-base=$${instloc} $${2}"))

seems to work, now qmake cannot find qtbase-dev/libexec/rcc

qmake PREFIX=/nix/store/yzp3mjp0dknrhsjxhwn12vk0cwgzfxpx-test NIX_OUTPUT_OUT=/nix/store/yzp3mjp0dknrhsjxhwn12vk0cwgzfxpx-test NIX_OUTPUT_DEV=/nix/store/yzp3mjp0dknrhsjxhwn12vk0cwgzfxpx-test NIX_OUTPUT_BIN=/nix/store/yzp3mjp0dknrhsjxhwn12vk0cwgzfxpx-test NIX_OUTPUT_QML=/nix/store/yzp3mjp0dknrhsjxhwn12vk0cwgzfxpx-test/lib/qt-6/qml NIX_OUTPUT_PLUGIN=/nix/store/yzp3mjp0dknrhsjxhwn12vk0cwgzfxpx-test/lib/qt-6/plugins CONFIG+=release LINUX_DISTRO=NixOS CONFIG+=WITH_I18N LRELEASE=/nix/store/v9lkwz407hsqskxcraz6qafblb0vwwv2-qttools-6.4.2-dev/bin/lrelease Info: creating stash file /build/qml-i18n/.qmake.stash sh: /nix/store/gskwk8h8rirfq27hmmm4963aillnziv4-qtbase-6.4.2-dev/mkspecs/features/rcc): not found

→ qtbase-dev/mkspecs/features/resources.prf - $$QMAKE_RCC

... success : )

LC_ALL=fr_FR.UTF-8 /nix/store/aa9i967q2s2xsqrq73y1vhhs9n360fcf-test/bin/qml-i18n

Bonjour

--- /a/nix/store/gskwk8h8rirfq27hmmm4963aillnziv4-qtbase-6.4.2-dev/mkspecs/features/qt_functions.prf    1970-01-01 01:00:01.000000000 +0100
+++ /b/nix/store/gskwk8h8rirfq27hmmm4963aillnziv4-qtbase-6.4.2-dev/mkspecs/features/qt_functions.prf    2023-02-05 20:31:48.293837562 +0100
@@ -96,6 +96,10 @@
             instloc = $$5
         }
         cmd = $$instloc/$$2
+# patch: allow passing absolute path to qtPrepareTool
+        !exists($${cmd}) {
+          cmd = $$system("realpath --relative-base=$${instloc} $${2}")
+        }
         exists($${cmd}.pl) {
             $${1}_EXE = $${cmd}.pl
             cmd = perl -w $$system_path($${cmd}.pl)

--- /a/nix/store/gskwk8h8rirfq27hmmm4963aillnziv4-qtbase-6.4.2-dev/mkspecs/features/lrelease.prf        1970-01-01 01:00:01.000000000 +0100
+++ /b/nix/store/v9lkwz407hsqskxcraz6qafblb0vwwv2-qttools-6.4.2-dev/mkspecs/features/lrelease.prf       2023-02-05 20:35:00.685916660 +0100
@@ -7,7 +7,7 @@
 # Otherwise, the .qm files are available in the build directory in LRELEASE_DIR.
 # They can also be automatically installed by setting QM_FILES_INSTALL_PATH.

-qtPrepareTool(QMAKE_LRELEASE, lrelease)
+qtPrepareTool(QMAKE_LRELEASE, /nix/store/v9lkwz407hsqskxcraz6qafblb0vwwv2-qttools-6.4.2-dev/bin/lrelease)

 isEmpty(LRELEASE_DIR): LRELEASE_DIR = .qm
 isEmpty(QM_FILES_RESOURCE_PREFIX): QM_FILES_RESOURCE_PREFIX = i18n

alternative: set $$5 = instloc = /nix/store/v9lkwz407hsqskxcraz6qafblb0vwwv2-qttools-6.4.2-dev/bin in the qtPrepareTool call

testing build with qtbase/examples/widgets/tools/i18n

qtbase-example-i18n.nix

```nix { pkgs ? import ./. {} }: with pkgs; with qt6; stdenv.mkDerivation { name = "qtbase-example-i18n"; src = qtbase.src; prePatch = '' cd examples/widgets/tools/i18n ''; # tries to install to ${qtbase.out}/examples/ installPhase = '' mkdir -p $out/bin cp i18n $out/bin ''; nativeBuildInputs = [ qmake qttools wrapQtAppsHook ]; buildInputs = [ qtbase ]; } ```

btw, im using my writable-nix-store for prototyping the patch

nix-build . -A qt6.qtbase.dev && mv result-dev qtbase-dev
nix-build . -A qt6.qttools.dev && mv result-dev qttools-dev
sudo ./writable-nix-store.js start
sudo mkdir qttools-dev/mkspecs/features
sudo mv qttools-dev/mkspecs/features/lrelease.prf qttools-dev/mkspecs/features
sudo $EDITOR qttools-dev/mkspecs/features/lrelease.prf
...

milahu commented 1 year ago

workaround: pkgs/desktops/lumina/lumina/default.nix

{
  qmakeFlags = [
    "LINUX_DISTRO=NixOS"
    "CONFIG+=WITH_I18N"
    "LRELEASE=${lib.getDev qttools}/bin/lrelease"
  ];

OPNA2608 commented 1 year ago

That is a Lumina-specific workaround because they build their localisation manually:

https://github.com/lumina-desktop/lumina/blob/3dbfc711785df2e9cb43a2a8e593638abaceb64d/src-qt5/OS-detect.pri#L113 https://github.com/lumina-desktop/lumina/blob/3dbfc711785df2e9cb43a2a8e593638abaceb64d/src-qt5/core/lumina-open/lumina-open.pro#L89-L90

Setting LRELEASE as a QMake flag does nothing for projects that do it via CONFIG += lrelease. And qtPrepareTool doesn't care about what's already stored in the target variable so setting QMAKE_LRELEASE doesn't work either.

milahu commented 1 year ago

fixed in ninja-edit

NickCao commented 1 year ago

So the fix is either porting the qt-5 patch, or patch mkspecs/features/lrelease.prf?

NickCao commented 1 year ago

I'm not seeing us introducing QMAKE_LRELEASE, this ought to be a upstream envvar. Should https://github.com/NixOS/nixpkgs/blob/216fbe09999764469066b1ccf05dcff45cf078e7/pkgs/development/libraries/qt-5/hooks/qttools-setup-hook.sh alone be enough? Edit: it's not working.

NickCao commented 1 year ago

I think it's actually logical to patch qtPrepareTool to locate binaries in PATH, or we would have to setup an additional search path, maybe QT_TOOL_PATH or something.

milahu commented 1 year ago

patch mkspecs/features/lrelease.prf

yes. move it to qttools.dev and patch the instloc

the only question is, how do we move it from qtbase to qttools. for example, first move it to ${qtbase.dev}/mkspecs/features/lrelease.prf.bak then copy it to ${qttools.dev}/mkspecs/features/lrelease.prf where we patch it

patch qtPrepareTool to locate binaries in PATH

no, this breaks the build-system in edge cases ... it defeats the whole purpose of having mkspecs files or the purpose of nix ("use multiple versions of packages")

introducing QMAKE_LRELEASE

its a workaround, which should be removed in other packages like lumina

patch the instloc

simple as

--- /a/nix/store/gskwk8h8rirfq27hmmm4963aillnziv4-qtbase-6.4.2-dev/mkspecs/features/lrelease.prf        1970-01-01 01:00:01.000000000 +0100
+++ /b/nix/store/v9lkwz407hsqskxcraz6qafblb0vwwv2-qttools-6.4.2-dev/mkspecs/features/lrelease.prf       2023-02-06 07:58:22.619611737 +0100
@@ -7,7 +7,7 @@
 # Otherwise, the .qm files are available in the build directory in LRELEASE_DIR.
 # They can also be automatically installed by setting QM_FILES_INSTALL_PATH.

-qtPrepareTool(QMAKE_LRELEASE, lrelease)
+qtPrepareTool(QMAKE_LRELEASE, lrelease, , , /nix/store/v9lkwz407hsqskxcraz6qafblb0vwwv2-qttools-6.4.2-dev/bin)

 isEmpty(LRELEASE_DIR): LRELEASE_DIR = .qm
 isEmpty(QM_FILES_RESOURCE_PREFIX): QM_FILES_RESOURCE_PREFIX = i18n

so no need to patch defineTest(qtPrepareTool) { in qtbase-dev/mkspecs/features/qt_functions.prf

parsing tools for automatic patching: qtbase-dev/mkspecs/features/qt_functions.prf → look for instloc. used in qtPrepareTool and qtPrepareLibExecTool

grep -r -h -o -E 'qtPrepareTool\([A-Z]+.*$' qtbase-dev/mkspecs/features/

``` qtPrepareTool(ANDROIDDEPLOYQT, androiddeployqt) qtPrepareTool(QMAKE_QGLTF, qgltf) qtPrepareTool(QMLPLUGINDUMP, $$qmlplugindump) qtPrepareTool(QMAKE_QMLTESTRUNNER, qmltestrunner) qtPrepareTool(QDOC, qdoc) qtPrepareTool(QMAKE_QDBUSXML2CPP, qdbusxml2cpp) qtPrepareTool(QMAKE_IDC, idc) qtPrepareTool(QMAKE_DUMPCPP, dumpcpp) qtPrepareTool(QMAKE_WINDEPLOYQT, windeployqt) qtPrepareTool(ANDROIDTESTRUNNER, androidtestrunner) qtPrepareTool(ANDROIDDEPLOYQT, androiddeployqt) ```

grep -r -h -o -E 'qtPrepareLibExecTool\([A-Z]+.*$' qtbase-dev/mkspecs/features/

``` qtPrepareLibExecTool(MOC_COLLECT_JSON, moc) qtPrepareLibExecTool(QHELPGENERATOR, qhelpgenerator) qtPrepareLibExecTool(QTATTRIBUTIONSSCANNER, qtattributionsscanner) qtPrepareLibExecTool(QMAKE_RCC, rcc, _DEP) qtPrepareLibExecTool(QMLIMPORTSCANNER, qmlimportscanner, , system) qtPrepareLibExecTool(QMAKE_SYNCQT, syncqt, , system) qtPrepareLibExecTool(QMAKE_QTWAYLANDSCANNER, qtwaylandscanner) qtPrepareLibExecTool(QMAKE_MOC, moc) qtPrepareLibExecTool(QMAKE_QLALR, qlalr) qtPrepareLibExecTool(QMAKE_UIC, uic, _DEP) qtPrepareLibExecTool(QMAKE_TRACEGEN, tracegen) ```

qtPrepareTool(QMLPLUGINDUMP, $$qmlplugindump)

    qmlplugindump = qmlplugindump
    qtPrepareTool(QMLPLUGINDUMP, $$qmlplugindump)

so ... in qtbase postInstall

disableBrokenMkspecsFiles() {
  # check if a tool is installed in the expected location
  # if the tool is not installed, disable the mkspecs file
  # by moving it to *.bak
  local funcname_expected=$1
  local instloc_expected=$2
  local mkspecs_dir=$dev/mkspecs/features/
  while IFS=, read -r mkspecs_file funcname variable \
      default_tool suffix prepare instloc; do
    # debug
    #echo mkspecs_file=$mkspecs_file; echo funcname=$funcname
    #echo variable=$variable; echo default_tool=$default_tool
    #echo suffix=$suffix; echo prepare=$prepare; echo instloc=$instloc
    #echo tool_file=$instloc_expected/$default_tool
    # fix: $$qmlplugindump -> qmlplugindump
    if [[ ${default_tool:0:2} == '$$' ]]
      then default_tool=${default_tool:2}; fi
    local tool_file=$instloc_expected/$default_tool
    if ! [ -e $tool_file ]; then
      # tool is not installed in instloc_expected
      echo missing tool: $tool_file
      if [ -e $mkspecs_file ]; then
        # move the mkspecs file to *.bak
        echo disabling mkspecs file: $mkspecs_file
        #echo mv $mkspecs_file{,.bak} # debug
        #mv $mkspecs_file{,.bak} # TODO restore
      fi
    fi
  done < <(
    grep -r -o -E $funcname_expected'\([A-Z]+.*$' $mkspecs_dir |
    sed -E -e 's/\)$//' -e 's/, /,/g' -e 's/[:(]/,/g'
  )
}

disableBrokenMkspecsFiles qtPrepareTool $dev/bin

``` disabling mkspecs file: qtbase-dev/mkspecs/features/qgltf.prf disabling mkspecs file: qtbase-dev/mkspecs/features/qml_plugin.prf disabling mkspecs file: qtbase-dev/mkspecs/features/qmltestcase.prf disabling mkspecs file: qtbase-dev/mkspecs/features/qt_docs.prf disabling mkspecs file: qtbase-dev/mkspecs/features/lrelease.prf disabling mkspecs file: qtbase-dev/mkspecs/features/win32/idcidl.prf disabling mkspecs file: qtbase-dev/mkspecs/features/win32/dumpcpp.prf disabling mkspecs file: qtbase-dev/mkspecs/features/win32/windeployqt.prf disabling mkspecs file: qtbase-dev/mkspecs/features/lrelease.prf.bak ```

disableBrokenMkspecsFiles qtPrepareLibExecTool $dev/libexec

``` disabling mkspecs file: qtbase-dev/mkspecs/features/qt_docs.prf disabling mkspecs file: qtbase-dev/mkspecs/features/qt.prf disabling mkspecs file: qtbase-dev/mkspecs/features/qt_module_headers.prf disabling mkspecs file: qtbase-dev/mkspecs/features/wayland-scanner.prf ```

NickCao commented 1 year ago

This fix introduces a lot of laborious work, like finding the correct place to restore the disabled files. However if you like it done this way, I'm happy to review it.

NickCao commented 1 year ago

I'm thinking of another possibility: move all the mkspecs files to their own output, then craft a passthru derivation with fixed mkspecs. That way we work around the cyclic dependencies, and there's no need to move them around.

NixOS / nixpkgs