[FR] Improve build system

[barolo]

Current build requirements are already humongous and are only growing, allow disabling some components of the app,

[mkiol]

It is not as scary as it may seem at first glance :)

If you want to build this app from the sources, you can change the CMake options and decide which dependencies you want to build or just link to libraries already installed in your OS. By default almost all dependencies are built from the sources. To change this, pass -DBUILD_X=OFF to CMake (X - is an option name).

For example, to build version with only OpenCL GPU acceleration and without any Python deps, you should use the following:

cmake <dsnote root dir> \
        -DCMAKE_BUILD_TYPE=Release \
        -DWITH_DESKTOP=ON \
        -DWITH_PY=OFF \
        -DBUILD_WHISPERCPP_CUBLAS=OFF \
        -DBUILD_WHISPERCPP_HIPBLAS=OFF
        -DCMAKE_INSTALL_PREFIX=/usr/local

make
sudo make install

If you are on latest Ubuntu, you can try to build deb package following this instruction. PS: Make sure you disabled ROCm and CUDA in PKGBUILD if you don't need them.

cd <dsnote root dir>/deb
makedeb

[barolo]

I wrote this after trying unsuccessfuly ;) I tried to build a stripped down version, just with whisper CPU. And it's just not possible at the moment, as the build system will complain about some unrelated dependency down the line. Like Piper or Espeak, even though I've disabled building of them. What I want is to remove dependency on certain packages altogether.

[mkiol]

system will complain about some unrelated dependency down the line. Like Piper or Espeak, even though I've disabled building of them. What I want is to remove dependency on certain packages altogether.

I see. Indeed, you cannot disable any library. For example, Piper or Espeak are statically linked, so you can't build without them.

Let's leave this issue open. Maybe I'll come up with a solution in the future.

PS: If you looking for Whisper only, I recommend you to check whisper.cpp release page. They also publish a command line tool (main) for STT audio processing. It works very well.

[barolo]

system will complain about some unrelated dependency down the line. Like Piper or Espeak, even though I've disabled building of them. What I want is to remove dependency on certain packages altogether.

I see. Indeed, you cannot disable any library. For example, Piper or Espeak are statically linked, so you can't build without them.

Let's leave this issue open. Maybe I'll come up with a solution in the future.

PS: If you looking for Whisper only, I recommend you to check whisper.cpp release page. They also publish a command line tool (main) for STT audio processing. It works very well.

Yeah, I found several replacements for my needs already, and managed to get opencl acceleration working. Still, would love to have Show Note as a fronted, it's just too annoying to build atm, but i'll try again. I can imagine it also makes it near impossible to package for some distributions. Thanks for your work.

[barolo]

Currently I'm failing at

make[2]: *** [CMakeFiles/rhvoicedata_module.dir/build.make:74: rhvoicedata.tar.xz] Error 127
make[1]: *** [CMakeFiles/Makefile2:793: CMakeFiles/rhvoicedata_module.dir/all] Error 2```
what is rhvoicedata.tar.xz supossed to be?

[mkiol]

Currently I'm failing at

I can't reproduce this problem 🤔. Can you try with clean build or remove CMakeCache.txt from your build directory?

And use these CMake options:

cmake <dsnote root dir> \
        -DCMAKE_BUILD_TYPE=Release \
        -DWITH_DESKTOP=ON \
        -DWITH_PY=OFF \
        -DBUILD_WHISPERCPP_CUBLAS=OFF \
        -DBUILD_WHISPERCPP_HIPBLAS=OFF \
        -DCMAKE_INSTALL_PREFIX=/usr/local

(Obviously, you can tune CMAKE_INSTALL_PREFIX)

what is rhvoicedata.tar.xz supossed to be?

It is compressed configuration files needed by RHVoice. Most likely you don't need them since you are only interested in Whisper but it is mandatory build step right now.

[barolo]

I did fresh clone and used the cmake command you have provided, still the same error. That script from tools for packaging RHvoice, doesn't seem to be working.

[mkiol]

I sill don't know why you got this error but...

I the most recent commits, I've managed to move 'vosk' and 'rhvoice' to optional features. This means that existence of these libraries is checked dynamically during runtime. Now you can freely disable them in the build.

To test this, please sync latest commits and start fresh build (or just remove CMakeCache.txt in current build).

Updated list of options:

cmake <dsnote root dir> \
        -DCMAKE_BUILD_TYPE=Release \
        -DWITH_DESKTOP=ON \
        -DWITH_PY=OFF \
        -DBUILD_WHISPERCPP_CUBLAS=OFF \
        -DBUILD_WHISPERCPP_HIPBLAS=OFF \
        -DBUILD_VOSK=OFF \
        -DBUILD_RHVOICE=OFF \
        -DBUILD_RHVOICE_MODULE=OFF \
        -DCMAKE_INSTALL_PREFIX=/usr/local

[barolo]

Thank you, this improved things, but I think that it will happen with all of these scripts, now it stops at :

dsnote/tools/make_espeakdata_module.sh: cannot execute: required file not found
make[2]: *** [CMakeFiles/espeakdata_module.dir/build.make:74: espeakdata.tar.xz] Error 127
make[1]: *** [CMakeFiles/Makefile2:716: CMakeFiles/espeakdata_module.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

[mkiol]

😿

New fix. Sync to latest commit and use following options:

cmake <dsnote root dir> \
        -DCMAKE_BUILD_TYPE=Release \
        -DWITH_DESKTOP=ON \
        -DWITH_PY=OFF \
        -DBUILD_WHISPERCPP_CUBLAS=OFF \
        -DBUILD_WHISPERCPP_HIPBLAS=OFF \
        -DBUILD_VOSK=OFF \
        -DBUILD_RHVOICE=OFF \
        -DBUILD_RHVOICE_MODULE=OFF \
        -DDOWNLOAD_LIBSTT=OFF \
        -DBUILD_BERGAMOT=OFF \
        -DBUILD_ESPEAK_MODULE=OFF \
        -DCMAKE_INSTALL_PREFIX=/usr/local

This is the most minimalist build. Hopefully it will work.

[barolo]

managed to fix the scripts! it was enough to change #!/usr/bin/bash to #!/bin/sh. It's building atm

[mkiol]

Nice catch! Assuming that every system has bash is definitely not right. Awesome that you found this.

[barolo]

😿

New fix. Sync to latest commit and use following options:

cmake <dsnote root dir> \
        -DCMAKE_BUILD_TYPE=Release \
        -DWITH_DESKTOP=ON \
        -DWITH_PY=OFF \
        -DBUILD_WHISPERCPP_CUBLAS=OFF \
        -DBUILD_WHISPERCPP_HIPBLAS=OFF \
        -DBUILD_VOSK=OFF \
        -DBUILD_RHVOICE=OFF \
        -DBUILD_RHVOICE_MODULE=OFF \
        -DDOWNLOAD_LIBSTT=OFF \
        -DBUILD_BERGAMOT=OFF \
        -DBUILD_ESPEAK_MODULE=OFF \
        -DCMAKE_INSTALL_PREFIX=/usr/local

This is the most minimalist build. Hopefully it will work.

Had to enable bergamot or it would fail down the line. Now it fails with:

.../src/text_tools.cpp:13:10: fatal error: maddy/parser.h: No such file or directory
   13 | #include <maddy/parser.h>
      |          ^~~~~~~~~~~~~~~~
compilation terminated.
~~~

[barolo]

it seems that maddy includes aren't properly copied into build directory Edit: This corrected maddy.cmake works

set(maddy_source_url "https://github.com/progsource/maddy/archive/refs/tags/1.3.0.tar.gz")
set(maddy_checksum "561681f8c8d2b998c153cda734107a0bc1dea4bb0df69fd813922da63fa9f3e7")

ExternalProject_Add(maddy
    SOURCE_DIR ${external_dir}/maddy
    BINARY_DIR ${PROJECT_BINARY_DIR}/external/maddy
    INSTALL_DIR ${PROJECT_BINARY_DIR}/external
    URL ${maddy_source_url}
    URL_HASH SHA256=${maddy_checksum}
    CONFIGURE_COMMAND ""
    BUILD_COMMAND ""
    INSTALL_COMMAND cp -r ${external_dir}/maddy/include/maddy ${external_include_dir}/
)

list(APPEND deps maddy)

[barolo]

It seems that I'm close but now it fails on :

[ 95%] Built target dsnote_autogen
make[2]: *** No rule to make target 'external/lib/libqhotkey.a', needed by 'dsnote'.  Stop.
make[1]: *** [CMakeFiles/Makefile2:266: CMakeFiles/dsnote.dir/all] Error 2

Edit. Ok, fixed that too, it was the same error waswith rubberband, the build system placed them in ../lib64 instead of ./lib and make couldn't find it

Now it builds! but fails to load qml upon launch

E] 22:09:50.846404312.846 0x7f05935a7080 () - failed to create qml object

[barolo]

Error log:

[D] 22:22:41.139385902.139 0x7fb63d550080 () - version: 4.5.0-beta
[D] 22:22:41.139627772.139 0x7fb63d550080 parse_cpuinfo:117 - cpu flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
[D] 22:22:41.139776190.139 0x7fb63d550080 parse_cpuinfo:125 - cpuinfo: processor-count=16, flags=[avx, avx2, fma, f16c, ]
[D] 22:22:41.139846125.139 0x7fb63d550080 () - translation: "en_US"
[W] 22:22:41.139852561.139 0x7fb63d550080 () - failed to install translation
[D] 22:22:41.139856428.139 0x7fb63d550080 () - starting standalone app
[D] 22:22:41.140617745.140 0x7fb63d550080 () - app: net.mkiol dsnote
[D] 22:22:41.140631140.140 0x7fb63d550080 () - config location: "/home/greggy/.config"
[D] 22:22:41.140635596.140 0x7fb63d550080 () - data location: "/home/greggy/.local/share/net.mkiol/dsnote"
[D] 22:22:41.140638842.140 0x7fb63d550080 () - cache location: "/home/greggy/.cache/net.mkiol/dsnote"
[D] 22:22:41.140642328.140 0x7fb63d550080 () - settings file: "/home/greggy/.config/net.mkiol/dsnote/settings.conf"
[D] 22:22:41.140646297.140 0x7fb63d550080 () - platform: "wayland"
[D] 22:22:41.140659566.140 0x7fb63d550080 () - enforcing num threads: 0
[D] 22:22:41.155698844.155 0x7fb63d550080 () - supported audio input devices:
[D] 22:22:41.169346724.169 0x7fb63d550080 () - "default"
[D] 22:22:41.232606300.232 0x7fb63d550080 () - "pipewire"
[D] 22:22:41.270812272.270 0x7fb63d550080 () - "sysdefault:CARD=sofhdadsp"
[D] 22:22:41.270848662.270 0x7fb63d550080 () - "alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_6__source"
[D] 22:22:41.270853899.270 0x7fb63d550080 () - "alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_5__sink.monitor"
[D] 22:22:41.270857128.270 0x7fb63d550080 () - "alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_4__sink.monitor"
[D] 22:22:41.270860399.270 0x7fb63d550080 () - "alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp_3__sink.monitor"
[D] 22:22:41.270863600.270 0x7fb63d550080 () - "alsa_output.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__sink.monitor"
[D] 22:22:41.270866393.270 0x7fb63d550080 () - "alsa_input.pci-0000_00_1f.3-platform-skl_hda_dsp_generic.HiFi__hw_sofhdadsp__source"
[D] 22:22:41.276360416.276 0x7fb63d550080 () - starting service: app-standalone
[D] 22:22:41.276828099.276 0x7fb63d550080 () - mbrola dir: "/usr/local/share/dsnote/bin"
[D] 22:22:41.276843466.276 0x7fb63d550080 () - espeak dir: "/usr/local/share/dsnote/bin"
[D] 22:22:41.276871548.276 0x7fb627e00680 loop:79 - py executor loop started
[D] 22:22:41.278504837.278 0x7fb63d550080 () - module already unpacked: "rhvoicedata"
[D] 22:22:41.278526204.278 0x7fb63d550080 () - module already unpacked: "rhvoiceconfig"
[D] 22:22:41.280275298.280 0x7fb63d550080 () - module already unpacked: "espeakdata"
[D] 22:22:41.280504373.280 0x7fb63d550080 () - default stt model not found: "en"
[D] 22:22:41.280514297.280 0x7fb63d550080 () - default tts model not found: "en"
[D] 22:22:41.280517626.280 0x7fb63d550080 () - default mnt lang not found: "en"
[D] 22:22:41.280520089.280 0x7fb63d550080 () - new default mnt lang: "en"
[D] 22:22:41.280526350.280 0x7fb63d550080 () - service refresh status, new state: busy
[D] 22:22:41.280529545.280 0x7fb63d550080 () - service state changed: unknown => busy
[D] 22:22:41.280532414.280 0x7fb63d550080 () - delaying features availability
[D] 22:22:41.281869265.281 0x7fb63d550080 () - runtime prefix: "/run/media/greggy/1a4fd6d7-1f9d-42c6-9324-661804695013/y_fresh/dsnote"
[W] 22:22:41.281890511.281 0x7fb63d550080 () - can't find dir for path: "lib" "plugins"
[W] 22:22:41.281904930.281 0x7fb63d550080 () - can't find dir for path: "lib" "qml"
[W] 22:22:41.281916491.281 0x7fb63d550080 () - can't find dir for path: "lib" "qml/QtQuick/Controls.2"
[D] 22:22:41.282098763.282 0x7fb63d550080 () - available styles: ("Default", "Fusion", "Imagine", "Material", "Universal")
[D] 22:22:41.282158658.282 0x7fb63d550080 () - style paths: ("/usr/lib64/qt5/qml/QtQuick/Controls.2")
[D] 22:22:41.282164904.282 0x7fb63d550080 () - import paths: ("/run/media/greggy/1a4fd6d7-1f9d-42c6-9324-661804695013/y_fresh/dsnote/build", "qrc:/qt-project.org/imports", "/usr/lib64/qt5/qml")
[D] 22:22:41.282173336.282 0x7fb63d550080 () - library paths: ("/usr/lib64/qt5/plugins", "/run/media/greggy/1a4fd6d7-1f9d-42c6-9324-661804695013/y_fresh/dsnote/build")
[D] 22:22:41.282178145.282 0x7fb63d550080 () - using auto qt style
[W] 22:22:41.282182330.282 0x7fb63d550080 () - default qt style not found
[D] 22:22:41.282188001.282 0x7fb63d550080 () - don't forcing any qt style
[D] 22:22:41.284340224.284 0x7fb635000680 () - config version: 66 66
[D] 22:22:41.307963738.307 0x7fb635000680 () - models changed
[W] 22:22:41.586620745.586 0x7fb63d550080 () - QQmlApplicationEngine failed to load component
[W] 22:22:41.586644221.586 0x7fb63d550080 ():173 - qrc:/qml/main.qml:173:13: Type MainToolBar unavailable
[W] 22:22:41.586647927.586 0x7fb63d550080 ():10 - qrc:/qml/MainToolBar.qml:10:1: module "QtQuick.Dialogs" is not installed
[D] 22:22:41.590112043.590 0x7fb63d550080 () - default stt model not found: "en"
[D] 22:22:41.590121847.590 0x7fb63d550080 () - default tts model not found: "en"
[D] 22:22:41.590124919.590 0x7fb63d550080 () - default mnt lang not found: "en"
[D] 22:22:41.590126845.590 0x7fb63d550080 () - new default mnt lang: "en"
[D] 22:22:41.590130150.590 0x7fb63d550080 () - service refresh status, new state: busy
[D] 22:22:41.590132510.590 0x7fb63d550080 () - service refresh status, new state: busy
[E] 22:22:41.590163067.590 0x7fb63d550080 () - failed to create qml object
[D] 22:22:41.592083777.592 0x7fb63d550080 () - exiting

[barolo]

Solved the above by installing qtquickcontrols [not qtquikcontrols2] which are deprecated, cmake should check for that too. Now the app launches. Yay! Are there plans to move to Qt6?

[mkiol]

Thank you for all the tests and struggles!

managed to fix the scripts! it was enough to change #!/usr/bin/bash to #!/bin/sh.

To avoid these kind of problems I should re-write all shell script to CMake. This would be the most portable solution. When I just changed to #!/bin/sh, it gave me an error on my system :/

it seems that maddy includes aren't properly copied into build directory Edit: This corrected maddy.cmake works

Edit. Ok, fixed that too, it was the same error waswith rubberband, the build system placed them in ../lib64 instead of ./lib and make couldn't find it

Many thanks. Fixed in 8266698d5caf7311c9af8afd70601f2b1ef6b1a0.

Solved the above by installing qtquickcontrols [not qtquikcontrols2] which are deprecated, cmake should check for that too.

These are actually runtime dependencies. I don't think the build system should check for this. For instance, you might just want to build this program, but you're not interested in running it. These kind of deps should be handled by packaging system, like this. For sure, I need to document this better.

Now the app launches. Yay!

🎉

Are there plans to move to Qt6?

Yes I have. I'm waiting for the official release of Flatpak runtime with Qt6. Now, only Qt5 is available.

[barolo]

There are some bugs related to theming/environment (visible in the above log). Breeze theme and kf filepicker (just the ugly Qt one) are't working. Probably due to the assumptions of flatpak environment, and locations of /qml /plugins dirs.

[mkiol]

Probably due to the assumptions of flatpak environment, and locations of /qml /plugins dirs

Theming is a nightmare to me, but it should work fine. I've tested it on various systems.

Most likely you need to install additional QML plugins. For example, here is a list or required packages for Ubuntu. I'm guessing you need to install these packages (or equivalent packages in your distribution):

'qml-module-org-kde-qqc2breezestyle'
'qml-module-org-kde-qqc2desktopstyle'
'qml-module-org-kde-quickcharts'
'qml-module-qtquick-controls'
'qml-module-qtquick-dialogs'

[barolo]

Ah, missed this one qqc2desktopstyle. I already have it but I'm using plasma 6, pulling the version for 5 would pull bunch of annoying dependencies. I''ll wait for the Qt6 upgrade then ;)