[FEATURE] Nix CMake build process

[pbek]

As requested by @NickCao in https://github.com/NixOS/nixpkgs/pull/332556#issuecomment-2269875889, it would be great to build QOwnNotes with CMake for nixpkgs.

Todo

• [x] Make build process work with nix
• [x] Get translations working
• [x] Scripting engine works
• [x] Add CI process with CMake for Qt5
• [x] Add CI process with CMake for Qt6
• [x] Update config for nixpkgs

[pbek]

@Waqar144, I was trying to build QOwnNotes with Qt6 and CMake in https://github.com/pbek/QOwnNotes/blob/feature/nix-cmake/default.nix via https://github.com/pbek/QOwnNotes/blob/08c1657991e650ede70e950edf1b1ab3dabb5a07/Makefile#L16-L17

I got the error message:

CMake Error at libraries/botan/CMakeLists.txt:14 (find_package):
  By not providing "FindQt5.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Qt5", but
  CMake did not find one.

  Could not find a package configuration file provided by "Qt5" with any of
  the following names:

    Qt5Config.cmake
    qt5-config.cmake

  Add the installation prefix of "Qt5" to CMAKE_PREFIX_PATH or set "Qt5_DIR"
  to a directory containing one of the above files.  If "Qt5" provides a
  separate development package or SDK, be sure it has been installed.

Does this ring any bell for you?

[pbek]

Note that I tried to use:

https://github.com/pbek/QOwnNotes/blob/d307a417936278c95b4c908fe0759c12b6df1eba/default.nix#L47-L50

Have you ever built QOwnNotes with Qt6 and CMake? I only built it in Qt5 with CMake, as far as I remember...

[pbek]

QON_QT6_BUILD=ON got me one step further! 👍️

I'm now at:

CMake Error at libraries/botan/CMakeLists.txt:21 (find_package):
  By not providing "FindBotan2.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Botan2", but
  CMake did not find one.

  Could not find a package configuration file provided by "Botan2" with any
  of the following names:

    Botan2Config.cmake
    botan2-config.cmake

  Add the installation prefix of "Botan2" to CMAKE_PREFIX_PATH or set
  "Botan2_DIR" to a directory containing one of the above files.  If "Botan2"
  provides a separate development package or SDK, be sure it has been
  installed.

@NickCao, do you maybe have an idea to get around that in nix?

[NickCao]

Either we build botan2 with cmake, or locate botan2 with pkg-config instead.

[pbek]

Either we build botan2 with cmake, or locate botan2 with pkg-config instead.

@NickCao, thank you! The latter would be nice, I guess. I haven't used pkg-config before. I only found derivations including it, but I actually don't know how to use it. How do we locate botan2 with pkg-config?

[NickCao]

Strange, it seems that we are already doing that: https://github.com/pbek/QOwnNotes/blob/064310baf0efc8b1edf3c8c0ef4903cf52fab9db/cmake/FindBotan2.cmake

[NickCao]

Ah the cmake folder is missing from CMAKE_MODULE_PATH thus it wasn't used.

[pbek]

Which nix conjuration do we need to use? 😊

[NickCao]

Which nix conjuration do we need to use? 😊

Surprise!

diff --git a/default.nix b/default.nix
index 604f1b831..10c8b8e31 100644
--- a/default.nix
+++ b/default.nix
@@ -25,7 +25,7 @@ in
 stdenv.mkDerivation {
   inherit pname appname version;

-  src = builtins.path { path = ./src; name = "qownnotes"; };
+  src = builtins.path { path = ./.; name = "qownnotes"; };

   nativeBuildInputs = [
     cmake

[NickCao]

The line appending to CMAKE_MODULE_PATH is at https://github.com/pbek/QOwnNotes/blob/064310baf0efc8b1edf3c8c0ef4903cf52fab9db/CMakeLists.txt#L5

[pbek]

That was quick. 😂🤣 Thank you very much! I'll build and test for a while!

[pbek]

Meh, in the source archive on https://github.com/pbek/QOwnNotes/releases/download I only use the src folder. 😬 We will need some more trickery in https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/applications/office/qownnotes/default.nix, wouldn't we?

[pbek]

I'll build and test for a while!

And I'm getting an error after building:

Install the project...
-- Install configuration: "Release"
-- Installing: /nix/store/gg3y5q6j549x74ld4l3frkmgc2r9avqh-qttranslations-6.7.2/translations/QOwnNotes_en.qm
CMake Error at src/cmake_install.cmake:54 (file):
  file INSTALL cannot copy file "/build/qownnotes/build/src/QOwnNotes_en.qm"
  to
  "/nix/store/gg3y5q6j549x74ld4l3frkmgc2r9avqh-qttranslations-6.7.2/translations/QOwnNotes_en.qm":
  Permission denied.
Call Stack (most recent call first):
  cmake_install.cmake:47 (include)

make: *** [Makefile:100: install] Error 1
error: builder for '/nix/store/qs41bc4n38j6zap0ild1g1priwscgg5i-qownnotes-local-build.drv' failed with exit code 2;
       last 10 log lines:
       > CMake Error at src/cmake_install.cmake:54 (file):
       >   file INSTALL cannot copy file "/build/qownnotes/build/src/QOwnNotes_en.qm"
       >   to
       >   "/nix/store/gg3y5q6j549x74ld4l3frkmgc2r9avqh-qttranslations-6.7.2/translations/QOwnNotes_en.qm":
       >   Permission denied.
       > Call Stack (most recent call first):
       >   cmake_install.cmake:47 (include)
       >
       > 
       > make: *** [Makefile:100: install] Error 1
       For full logs, run 'nix log /nix/store/qs41bc4n38j6zap0ild1g1priwscgg5i-qownnotes-local-build.drv'.
make: *** [Makefile:17: nix-build] Fehler 1

[NickCao]

Meh, in the source archive on https://github.com/pbek/QOwnNotes/releases/download I only use the src folder. 😬 We will need some more trickery in https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/applications/office/qownnotes/default.nix, wouldn't we?

We can move the entire cmake folder under src (it's a single file anyway), and also move the line appending CMAKE_MODULE_PATH from the top-level CMakeLists.txt to the one inside src.

[NickCao]

I'll build and test for a while!

And I'm getting an error after building:

Install the project...
-- Install configuration: "Release"
-- Installing: /nix/store/gg3y5q6j549x74ld4l3frkmgc2r9avqh-qttranslations-6.7.2/translations/QOwnNotes_en.qm
CMake Error at src/cmake_install.cmake:54 (file):
  file INSTALL cannot copy file "/build/qownnotes/build/src/QOwnNotes_en.qm"
  to
  "/nix/store/gg3y5q6j549x74ld4l3frkmgc2r9avqh-qttranslations-6.7.2/translations/QOwnNotes_en.qm":
  Permission denied.
Call Stack (most recent call first):
  cmake_install.cmake:47 (include)

make: *** [Makefile:100: install] Error 1
error: builder for '/nix/store/qs41bc4n38j6zap0ild1g1priwscgg5i-qownnotes-local-build.drv' failed with exit code 2;
       last 10 log lines:
       > CMake Error at src/cmake_install.cmake:54 (file):
       >   file INSTALL cannot copy file "/build/qownnotes/build/src/QOwnNotes_en.qm"
       >   to
       >   "/nix/store/gg3y5q6j549x74ld4l3frkmgc2r9avqh-qttranslations-6.7.2/translations/QOwnNotes_en.qm":
       >   Permission denied.
       > Call Stack (most recent call first):
       >   cmake_install.cmake:47 (include)
       >
       > 
       > make: *** [Makefile:100: install] Error 1
       For full logs, run 'nix log /nix/store/qs41bc4n38j6zap0ild1g1priwscgg5i-qownnotes-local-build.drv'.
make: *** [Makefile:17: nix-build] Fehler 1

Guess we have to set QT_TRANSLATIONS_DIR in cmakeFlags

[pbek]

Now I got to:

CMake Error at src/libraries/md4c/src/cmake_install.cmake:115 (file):
  file INSTALL cannot find "/build/qownnotes/build/src/md4c.pc": No such file
  or directory.
Call Stack (most recent call first):
  src/libraries/md4c/cmake_install.cmake:47 (include)
  src/cmake_install.cmake:118 (include)
  cmake_install.cmake:47 (include)

🤔

[NickCao]

I believe this can be an upstream bug: https://github.com/qownnotes/md4c/blob/c64ee9ab326c53962b5bd8cca98c086461bbdd6b/src/CMakeLists.txt#L40, try changing CMAKE_BINARY_DIR to CMAKE_CURRENT_BINARY_DIR?

[Waqar144]

I have a cmake patch that fixes the install, will push it soon

[Waqar144]

Fixes the install == makes it work the same as qmake

So that it can be a drop in replacement

[pbek]

Thank you, @Waqar144

[pbek]

@NickCao, I now get:

evaling implicit 'postInstall' string hook
/nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/bin/QOwnNotes: error while loading shared libraries: libmd4c-html.so.0: cannot open shared object file: No such file or directory
/nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/bin/QOwnNotes: error while loading shared libraries: libmd4c-html.so.0: cannot open shared object file: No such file or directory
/nix/store/x08g3xj39jfv78gg950lf9r48hv16kzs-xvfb-run-1+g87f6705/bin/.xvfb-run-wrapped: line 186: kill: (1979) - No such process
installShellCompletion: error: installed shell completion file `/nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/bash-completion/completions/QOwnNotes.bash' does not exist or has zero size
/nix/store/5r0df66ikad3xw06azlqvswcvncll8wa-stdenv-linux/setup: line 193: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/13pwsk97b6ld8nvgcxpdpj28ssf4n001-qownnotes-local-build.drv' failed with exit code 1;
       last 10 log lines:
       > -- Installing: /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/qt6/translations/QOwnNotes_bs.qm
       > -- Installing: /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/qt6/translations/QOwnNotes_mk.qm
       > -- Installing: /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/qt6/translations/QOwnNotes_ko.qm
       > -- Installing: /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/qt6/translations/QOwnNotes_et.qm
       > evaling implicit 'postInstall' string hook
       > /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/bin/QOwnNotes: error while loading shared libraries: libmd4c-html.so.0: cannot open shared object file: No such file or directory
       > /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/bin/QOwnNotes: error while loading shared libraries: libmd4c-html.so.0: cannot open shared object file: No such file or directory
       > /nix/store/x08g3xj39jfv78gg950lf9r48hv16kzs-xvfb-run-1+g87f6705/bin/.xvfb-run-wrapped: line 186: kill: (1979) - No such process
       > installShellCompletion: error: installed shell completion file `/nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/bash-completion/completions/QOwnNotes.bash' does not exist or has zero size
       > /nix/store/5r0df66ikad3xw06azlqvswcvncll8wa-stdenv-linux/setup: line 193: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/13pwsk97b6ld8nvgcxpdpj28ssf4n001-qownnotes-local-build.drv'.
make: *** [Makefile:17: nix-build] Error 1

[Waqar144]

@pbek

Are you running CMake from the top level directory? Its a must, its not possible to run it from the src directory.

Also, you can change the install prefix when running cmake e.g,. cmake -DCMAKE_INSTALL_PREFIX=/path/to/installation

[pbek]

Are you running CMake from the top level directory? Its a must, its not possible to run it from the src directory.

Yes, I do. https://github.com/pbek/QOwnNotes/blob/f39a558e449f72973b72b40c7f7d7fa2f1a83e27/default.nix#L28

But this will be a problem in the future, because the release archive, like https://github.com/pbek/QOwnNotes/releases/download/v24.8.4/qownnotes-24.8.4.tar.xz is only containing the content of the src folder, because that's all that was needed to build it with qmake.

So is there a way to make it work with the src folder only?

[Waqar144]

That should be easy to change.

[NickCao]

@NickCao, I now get:

evaling implicit 'postInstall' string hook
/nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/bin/QOwnNotes: error while loading shared libraries: libmd4c-html.so.0: cannot open shared object file: No such file or directory
/nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/bin/QOwnNotes: error while loading shared libraries: libmd4c-html.so.0: cannot open shared object file: No such file or directory
/nix/store/x08g3xj39jfv78gg950lf9r48hv16kzs-xvfb-run-1+g87f6705/bin/.xvfb-run-wrapped: line 186: kill: (1979) - No such process
installShellCompletion: error: installed shell completion file `/nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/bash-completion/completions/QOwnNotes.bash' does not exist or has zero size
/nix/store/5r0df66ikad3xw06azlqvswcvncll8wa-stdenv-linux/setup: line 193: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/13pwsk97b6ld8nvgcxpdpj28ssf4n001-qownnotes-local-build.drv' failed with exit code 1;
       last 10 log lines:
       > -- Installing: /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/qt6/translations/QOwnNotes_bs.qm
       > -- Installing: /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/qt6/translations/QOwnNotes_mk.qm
       > -- Installing: /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/qt6/translations/QOwnNotes_ko.qm
       > -- Installing: /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/qt6/translations/QOwnNotes_et.qm
       > evaling implicit 'postInstall' string hook
       > /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/bin/QOwnNotes: error while loading shared libraries: libmd4c-html.so.0: cannot open shared object file: No such file or directory
       > /nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/bin/QOwnNotes: error while loading shared libraries: libmd4c-html.so.0: cannot open shared object file: No such file or directory
       > /nix/store/x08g3xj39jfv78gg950lf9r48hv16kzs-xvfb-run-1+g87f6705/bin/.xvfb-run-wrapped: line 186: kill: (1979) - No such process
       > installShellCompletion: error: installed shell completion file `/nix/store/382z0p54hfdkq2rhriqr93mhbl9cpcvh-qownnotes-local-build/share/bash-completion/completions/QOwnNotes.bash' does not exist or has zero size
       > /nix/store/5r0df66ikad3xw06azlqvswcvncll8wa-stdenv-linux/setup: line 193: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/13pwsk97b6ld8nvgcxpdpj28ssf4n001-qownnotes-local-build.drv'.
make: *** [Makefile:17: nix-build] Error 1

This could mean md4c has been installed to the wrong directory (or not installed at all). Again we should preferably use md4c from nixpkgs if possible.

[Waqar144]

We can't use system md4c, we just need to force static linking or just compile md4c into QOwnNotes like qmake does.

[pbek]

Plus, we are using https://github.com/qownnotes/md4c. 😉

[pbek]

@NickCao, I tried to build for qt5 as a nix derivation and got the same error. Building and running the app with CMake in CLion (under NixOS) works tho (running with wrapQtAppsHook).

[NickCao]

@NickCao, I tried to build for qt5 as a nix derivation and got the same error. Building and running the app with CMake in CLion (under NixOS) works tho (running with wrapQtAppsHook).

I checked the build output, and md4c is indeed not installed.

Possible fix:

For md4c (don't know how to fix installation of these .pc files so dropped them altogether):

diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 767c259..4800e97 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -37,7 +37,6 @@ install(
     PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
     INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
 )
-install(FILES ${CMAKE_BINARY_DIR}/src/md4c.pc DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)

 install(
     TARGETS md4c-html
@@ -47,7 +46,5 @@ install(
     RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
     PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
 )
-install(FILES ${CMAKE_BINARY_DIR}/src/md4c-html.pc DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)

 install(EXPORT md4cConfig DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/md4c/ NAMESPACE md4c::)

For qownnotes

--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -29,8 +29,7 @@ include(libraries/sonnet/src/plugins/hunspell/hunspell/CMakeLists.txt)

 # noneed for md2html
 set(BUILD_MD2HTML_EXECUTABLE FALSE)
-set(BUILD_SHARED_LIBS OFF)
-add_subdirectory(libraries/md4c EXCLUDE_FROM_ALL)
+add_subdirectory(libraries/md4c)

 add_executable(QOwnNotes "")

[pbek]

With #3080 of @Waqar144 the whole build process now went through! 🥳🎉 The translations and the script engine also works!

[pbek]

https://github.com/pbek/QOwnNotes/blob/475afebce832a8d48d3f12a3668611912c204ff2/default.nix was used for building. I hope this will also work with a release archive and on Darwin.

[Waqar144]

I think we should try to move all Qt6 builds to cmake, it has much better support (and documentation) everywhere. Once Qt6 is in place, one can think about moving the OBS builders and others slowly.

[pbek]

There are so many build services QOwnNotes deploys to. 🙈 😁

[sfatula]

https://github.com/pbek/QOwnNotes/blob/475afebce832a8d48d3f12a3668611912c204ff2/default.nix was used for building. I hope this will also work with a release archive and on Darwin.

Well, hopefully you don't mind me interjecting, and if you do, I will bow out. Actually note that xvfb-run does not exist in nix for darwin, either chip type, unless I am missing something as my nix build got an error for that. Looks like linux only. I'm trying on x86_64 (which is unnecessary since I could just download it), though I will have a M1 Mac shortly (delayed by USPS). I suppose I could try and build from source on the M1 but saw this case so tried it.

[pbek]

Thank you for pointing that out! 👍🏻 I guess that discussion needs to be taken directly to nixpkgs or in a separate issue. It has nothing to do with CMake.

Is there any alternative? Something to put in a PR for:

https://github.com/NixOS/nixpkgs/blob/8de2d5efd579346ed6836ec34d738f9377e9a46c/pkgs/applications/office/qownnotes/default.nix#L55-L60

I'll do a PR for the CMake transition with the next release of QOwnNotes.

[pbek]

@NickCao, I created https://github.com/NixOS/nixpkgs/pull/334463.

[pbek]

24.8.5

• The CMake build process was adapted to be production ready and to be used in Nix