Switching to nlohmann_json breaks OpenEmbedded builds

WanWizard commented 3 weeks ago

After https://github.com/windytan/redsea/commit/81eea4b26b60fecbc6496fbac924ebd8816819a6, redsea can no longer be built by OpenEmbedded, because it's meson build class uses --wrap-mode nodownload.

Log data follows:
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['/data/oe/kirkstone/meta-openpli/recipes-multimedia/liquid-dsp/liquid-dsp.bb:do_populate_sysroot', '/data/oe/kirkstone/openembedded-core/meta/recipes-core/glibc/glibc_2.35.bb:do_populate_sysroot', '/data/oe/kirkstone/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_11.4.bb:do_populate_sysroot', '/data/oe/kirkstone/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_11.4.bb:do_populate_sysroot', '/data/oe/kirkstone/openembedded-core/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_populate_sysroot', '/data/oe/kirkstone/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_populate_sysroot', '/data/oe/kirkstone/openembedded-core/meta/recipes-multimedia/libsndfile/libsndfile1_1.0.31.bb:do_populate_sysroot', 'virtual:native:/data/oe/kirkstone/meta-openpli/recipes-devtools/meson/meson_1.3.1.bb:do_populate_sysroot', 'virtual:native:/data/oe/kirkstone/openembedded-core/meta/recipes-core/gettext/gettext_0.21.bb:do_populate_sysroot', 'virtual:native:/data/oe/kirkstone/openembedded-core/meta/recipes-devtools/ninja/ninja_1.10.2.bb:do_populate_sysroot', 'virtual:native:/data/oe/kirkstone/openembedded-core/meta/recipes-devtools/patch/patch_2.7.6.bb:do_populate_sysroot', 'virtual:native:/data/oe/kirkstone/openembedded-core/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb:do_populate_sysroot', 'virtual:native:/data/oe/kirkstone/openembedded-core/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', 'virtual:native:/data/oe/kirkstone/openembedded-core/meta/recipes-devtools/python/python3_3.10.14.bb:do_populate_sysroot']
| NOTE: Installed into sysroot: []
| NOTE: Skipping as already exists in sysroot: ['liquid-dsp', 'glibc', 'gcc-cross-arm', 'gcc-runtime', 'qemu-native', 'quilt-native', 'libsndfile1', 'meson-native', 'gettext-native', 'ninja-native', 'patch-native', 'pkgconfig-native', 'pseudo-native', 'python3-native', 'xz-native', 'glib-2.0-native', 'zlib-native', 'attr-native', 'libtool-native', 'python3-installer-native', 'python3-wheel-native', 'python3-setuptools-native', 'linux-libc-headers', 'libgcc', 're2c-native', 'texinfo-dummy-native', 'gettext-minimal-native', 'openssl-native', 'libtirpc-native', 'ncurses-native', 'readline-native', 'sqlite3-native', 'libffi-native', 'bzip2-native', 'gdbm-native', 'util-linux-libuuid-native', 'libnsl2-native', 'flac', 'alsa-lib', 'libogg', 'libvorbis', 'binutils-cross-arm', 'gnu-config-native', 'mpfr-native', 'gmp-native', 'flex-native', 'libmpc-native', 'zstd-native', 'util-linux-native', 'libpcre-native', 'python3-flit-core-native', 'unzip-native', 'perl-native', 'm4-native', 'libcap-ng-native', 'libpcre2-native', 'make-native']
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing shell function do_configure
| NOTE: Executing meson ...
| DEPRECATION: "pkgconfig" entry is deprecated and should be replaced by "pkg-config"
| DEPRECATION: "pkgconfig" entry is deprecated and should be replaced by "pkg-config"
| The Meson build system
| Version: 1.3.1
| Source dir: /data/oe/kirkstone/build/tmp/work/armv7ahf-neon-oe-linux-gnueabi/redsea/gitAUTOINC+12819d0275-r0/git
| Build dir: /data/oe/kirkstone/build/tmp/work/armv7ahf-neon-oe-linux-gnueabi/redsea/gitAUTOINC+12819d0275-r0/build
| Build type: cross build
| Project name: redsea
| Project version: 1.0
| C++ compiler for the host machine: arm-oe-linux-gnueabi-g++ -march=armv7-a -mfpu=neon -mfloat-abi=hard --sysroot=/data/oe/kirkstone/build/tmp/work/armv7ahf-neon-oe-linux-gnueabi/redsea/gitAUTOINC+12819d0275-r0/recipe-sysroot (gcc 11.4.0 "arm-oe-linux-gnueabi-g++ (GCC) 11.4.0")
| C++ linker for the host machine: arm-oe-linux-gnueabi-g++ -march=armv7-a -mfpu=neon -mfloat-abi=hard --sysroot=/data/oe/kirkstone/build/tmp/work/armv7ahf-neon-oe-linux-gnueabi/redsea/gitAUTOINC+12819d0275-r0/recipe-sysroot ld.bfd 2.38.20220708
| C++ compiler for the build machine: g++ (gcc 11.4.1 "g++ (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)")
| C++ linker for the build machine: g++ ld.bfd 2.35.2-43
| Build machine cpu family: x86_64
| Build machine cpu: x86_64
| Host machine cpu family: arm
| Host machine cpu: arm
| Target machine cpu family: arm
| Target machine cpu: arm
| Configuring config.h using configuration
| Compiler for C++ supports arguments -Wno-unknown-pragmas: YES
| ../git/meson.build:29: WARNING: Consider using the built-in option for language standard version instead of using "-std=c++14".
| Found pkg-config: YES (/data/oe/kirkstone/build/tmp/work/armv7ahf-neon-oe-linux-gnueabi/redsea/gitAUTOINC+12819d0275-r0/recipe-sysroot-native/usr/bin/pkg-config) 0.29.2
| Run-time dependency sndfile found: YES 1.0.31
| Found CMake: NO
| Run-time dependency nlohmann_json found: NO (tried pkgconfig and cmake)
| Looking for a fallback subproject for the dependency nlohmann_json
| 
| ../git/meson.build:42:7: ERROR: Automatic wrap-based subproject downloading is disabled
| 
| A full log can be found at /data/oe/kirkstone/build/tmp/work/armv7ahf-neon-oe-linux-gnueabi/redsea/gitAUTOINC+12819d0275-r0/build/meson-logs/meson-log.txt
| WARNING: Running the setup command as `meson [options]` instead of `meson setup [options]` is ambiguous and deprecated.
| ERROR: meson failed

windytan commented 3 weeks ago

Hi! Thank you so much for the report. Glad to hear redsea being used in a new platform. Unfortunately I'm not familiar with OpenEmbedded; can you explain to me

How does one usually install libraries like nlohmann-json onto OpenEmbedded?
Do you have to compile them yourself, normally?
Is there a packaging system?
What would be an ideal solution to this problem?

windytan commented 2 weeks ago

Oh, just one more question @WanWizard : you mentioned that it 'breaks the build' - does this mean there is a build pipeline for building redsea on OpenEmbedded somewhere? Can I see the whole pipeline and set of commands?

Thanks in advance for your patience answering these!

WanWizard commented 2 weeks ago

We ( https://openpli.org https://github.com/openpli ) use OpenEmbedded as OS for DVB compliant set top boxes, on which we run a custom application to provide all A/V functionality. Amongst which is support for RTL-SDR USB audio.

OpenEmbedded uses bitbake ( https://github.com/openembedded/bitbake ) to build, which is a very complex "make" system, which uses python classes to support a variety of build environment, like meson ( https://github.com/openembedded/openembedded-core/blob/kirkstone/meta/classes/meson.bbclass ), so in the actual build recipe, you only have to define you want to use meson, and bitbake does the rest: https://github.com/OpenPLi/openpli-oe-core/blob/Kirkstone/meta-openpli/recipes-multimedia/redsea/redsea.bb

The bitbake meson.bbclass hardcoded defines "--wrap-mode", I'm currently looking at ways to override it.

WanWizard commented 2 weeks ago

Managed to override the meson.bbclass and removed the "nodownload", but now got stuck with the download failing:

Looking for a fallback subproject for the dependency nlohmann_json
Downloading nlohmann_json source from https://github.com/nlohmann/json/releases/download/v3.11.2/include.zip
<urlopen error [Errno -2] Name or service not known>
WARNING: failed to download with error: could not get https://github.com/nlohmann/json/releases/download/v3.11.2/include.zip is the internet available?. Trying after a delay...
<urlopen error [Errno -2] Name or service not known>
WARNING: failed to download with error: could not get https://github.com/nlohmann/json/releases/download/v3.11.2/include.zip is the internet available?. Trying after a delay...
<urlopen error [Errno -2] Name or service not known>
WARNING: failed to download with error: could not get https://github.com/nlohmann/json/releases/download/v3.11.2/include.zip is the internet available?. Trying after a delay...
<urlopen error [Errno -2] Name or service not known>
WARNING: failed to download with error: could not get https://github.com/nlohmann/json/releases/download/v3.11.2/include.zip is the internet available?. Trying after a delay...
<urlopen error [Errno -2] Name or service not known>
WARNING: failed to download with error: could not get https://github.com/nlohmann/json/releases/download/v3.11.2/include.zip is the internet available?. Trying after a delay...
<urlopen error [Errno -2] Name or service not known>
A fallback URL could be specified using source_fallback_url key in the wrap file

../git/meson.build:42:7: ERROR: could not get https://github.com/nlohmann/json/releases/download/v3.11.2/include.zip is the internet available?

which is rubbish, as it can clone the redsea repo without problems,. and also a wget of that URL works fine.

More work to do...

windytan commented 2 weeks ago

Just a thought, would it be possible to install the nlohmann-json library via the same mechanism that provides the liquid and sndfile dependencies? In that case, meson would find the library and not fall back to its own http fetch.

Meson has its own http fallback here only because I provided a "wrap file" for nlhomann-json, just in case it's not available in all distros.

WanWizard commented 2 weeks ago

Always handy to have a second pair of eyes look at it. Creating the dependency at bitbake level, like sndfile, fixed the build, thanks !

Still puzzled though why python wasn't able to resolve github and download the zip though...

windytan / redsea

Switching to nlohmann_json breaks OpenEmbedded builds #112