fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.64k stars 1.54k forks source link

BitBake compilation with Lua scripting support failing #8559

Closed lizziemac closed 2 months ago

lizziemac commented 5 months ago

Bug Report

Describe the bug All of the .bb examples I've found have had the -DFLB_LUAJIT and -DFLB_FILTER_LUA defines set to Off. When I set them both to On, I see the below linker error.

/home/build/gtmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native/usr/bin/cmake -E cmake_link_script CMakeFiles/minilua.dir/link.txt --verbose=1
/home/build/gtmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-gcc   -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native=  --sysroot=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native= -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fmacro-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native=  -Wl,-z,relro,-z,now -rdynamic CMakeFiles/minilua.dir/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/luajit-3065c9/src/host/minilua.c.o -o minilua 
/home/build/gtmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/../../libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.3.0/ld: yes
checking mach-o/dyld.h usability... [ 16%] Building C object lib/chunkio/src/CMakeFiles/chunkio-static.dir/cio_file_unix.c.o
cd /home/build/gtmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/build/lib/chunkio/src && /home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-gcc -DCIO_HAVE_BACKEND_FILESYSTEM -DCIO_HAVE_FALLOCATE -DCIO_HAVE_GETPAGESIZE -DCIO_HAVE_POSIX_FALLOCATE -DFLB_HAVE_CHUNK_TRACE -DFLB_HAVE_IN_STORAGE_BACKLOG -DFLB_HAVE_PARSER -DFLB_HAVE_RECORD_ACCESSOR -DFLB_HAVE_STREAM_PROCESSOR -DFLB_HAVE_SYS_WAIT_H -DFLB_SYSTEM_LINUX -DMPACK_EXTENSIONS=1 -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/fluent-otel-proto/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/fluent-otel-proto/proto_c -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/cfl/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/cfl/lib/xxhash -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/flb_libco -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/rbtree -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/msgpack-c/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/chunkio/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/build/lib/chunkio/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/luajit-3065c9/src -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/build/lib/luajit-cmake -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/monkey/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/monkey/include/monkey -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/sqlite-amalgamation-3440200 -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/mpack-amalgamation-1.1.1/src -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/miniz -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/onigmo -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/snappy-fef67ac -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/cmetrics/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/ctraces/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/c-ares-1.24.0/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/lwrb/lwrb/src/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/build/lib/c-ares-1.24.0 -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/build/lib/jansson-e23f558/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/build/lib/cmetrics -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/build/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/build/lib/monkey/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/build/lib/monkey/include/monkey -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/librdkafka-2.3.0/src -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/tutf8e/include -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/chunkio/deps -I/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/chunkio/deps/monkey/include -mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native=   --sysroot=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot  -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0=/usr/src/debug/fluent-bit/2.2.2-r0                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native=  -Wall -D__FLB_FILENAME__=__FILE__ -Wall  -D__FILENAME__=__FILE__ -g -fPIC -MD -MT lib/chunkio/src/CMakeFiles/chunkio-static.dir/cio_file_unix.c.o -MF CMakeFiles/chunkio-static.dir/cio_file_unix.c.o.d -o CMakeFiles/chunkio-static.dir/cio_file_unix.c.o -c /home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/chunkio/src/cio_file_unix.c
CMakeFiles/minilua.dir/home/build/gtmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/luajit-3065c9/src/host/minilua.c.o: in function `luaV_execute':
/usr/src/debug/fluent-bit/2.2.2-r0/git/lib/luajit-3065c9/src/host/minilua.c:5046: undefined reference to `pow'
/home/build/gtmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/../../libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.3.0/ld: CMakeFiles/minilua.dir/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/luajit-3065c9/src/host/minilua.c.o: in function `Arith':
/usr/src/debug/fluent-bit/2.2.2-r0/git/lib/luajit-3065c9/src/host/minilua.c:4928: undefined reference to `pow'
/home/build/gtmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/../../libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/11.3.0/ld: CMakeFiles/minilua.dir/home/build/amba-sdk/ambarella/out/yocto_out/cv28m_cashewnut/tmp/work/cortexa53-crypto-poky-linux/fluent-bit/2.2.2-r0/git/lib/luajit-3065c9/src/host/minilua.c.o: in function `constfolding':
/usr/src/debug/fluent-bit/2.2.2-r0/git/lib/luajit-3065c9/src/host/minilua.c:3498: undefined reference to `pow'
collect2: error: ld returned 1 exit status
fluent-bit_2.2.2.bb # Fluent Bit - Yocto / Bitbake # ============================ # The following Bitbake package the latest Fluent Bit stable release. SUMMARY = "Fast Log processor and Forwarder" DESCRIPTION = "Fluent Bit is a data collector, processor and \ forwarder for Linux. It supports several input sources and \ backends (destinations) for your data. \ " HOMEPAGE = "http://fluentbit.io" BUGTRACKER = "https://github.com/fluent/fluent-bit/issues" LICENSE = "Apache-2.0" LIC_FILES_CHKSUM = "file://LICENSE;md5=2ee41112a44fe7014dce33e26468ba93" SECTION = "net" PR = "r0" PV = "2.2.2" SRCREV = "eeea396e88da26f586a7cc39df8017ab97f06939" SRC_URI = "git://github.com/fluent/fluent-bit.git;protocol=https;nobranch=1" S = "${WORKDIR}/git" DEPENDS = "zlib bison-native flex-native" DEPENDS += " libyaml curl " RDEPENDS:${PN} += " libcurl " INSANE_SKIP:${PN}-dev += "dev-elf" # Use CMake 'Unix Makefiles' generator OECMAKE_GENERATOR ?= "Unix Makefiles" # Fluent Bit build options # ======================== # Host related setup EXTRA_OECMAKE += "-DGNU_HOST=${HOST_SYS} " # Enable LuaJIT and filter_lua support (for redacting sensitive data) EXTRA_OECMAKE += "-DFLB_LUAJIT=On -DFLB_FILTER_LUA=On " # Disable Library and examples EXTRA_OECMAKE += "-DFLB_SHARED_LIB=Off -DFLB_EXAMPLES=Off " # Systemd support (optional) DEPENDS += "systemd" EXTRA_OECMAKE += "-DFLB_IN_SYSTEMD=On " # Kafka Output plugin (disabled by default): note that when # enabling Kafka output plugin, the backend library librdkafka # requires 'openssl' as a dependency. # # DEPENDS += "openssl " # EXTRA_OECMAKE += "-DFLB_OUT_KAFKA=On " inherit cmake systemd SYSTEMD_SERVICE:${PN} = "fluent-bit.service" TARGET_CC_ARCH:append = " ${SELECTED_OPTIMIZATION}"

To Reproduce

Expected behavior When compiling fluentbit with Lua support, it succeeds, and I can add Lua filters like the following to my setup:

[FILTER]
    Name                       lua
    Match                      journal
    call                       do_a_thing
    script                     /etc/fluent-bit/scripts/my-script.lua

Your Environment

Additional context

patrick-stephens commented 5 months ago

Yes, I don't think LUA is supported due to some additional requirements. Yocto/Bitbake is currently best effort support - we can't test it in CI at the moment to verify anything or officially release it.

patrick-stephens commented 5 months ago

@tarruda may be able to help with the specific requirements of building LUA but my understanding is it was not possible on Bitbake at the time it was last tried. If it is then feel free to submit a PR.

thbonotto commented 5 months ago

In the file: lib/luajit-cmake/host/minilua/CMakeLists.txt

There is an if to add LIBM:

find_library(LIBM_LIBRARIES NAMES m)
...
if(LIBM_LIBRARIES)
  target_link_libraries(minilua ${LIBM_LIBRARIES})
endif()

But it does not fail if it can't find LIBM_LIBRARIES (find_library(LIBM_LIBRARIES NAMES m))

I tried to inherit pkg-config in the recipe but it still couldn't find it.

LIBM is a very primitive library, so I tried to force the linking

- if(LIBM_LIBRARIES)
-  target_link_libraries(minilua ${LIBM_LIBRARIES})
-endif()
+ target_link_libraries(minilua libm.so)

And it worked for me.

patrick-stephens commented 5 months ago

I think that is for the luajit project though so it would need patching there if we wanted to import it but hopefully gives @lizziemac some info. A docs PR on it would be good too if it works: https://github.com/fluent/fluent-bit-docs/blob/master/installation/yocto-embedded-linux.md

Be good to update the minimum version there as well.

lizziemac commented 5 months ago

I was able to build with the above patch (applied to the fluent-bit recipe) and also by setting the number of parallel processes to 1, because there was an issue with luajit compiling dependencies after they were needed. I will look into creating a PR

github-actions[bot] commented 2 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

github-actions[bot] commented 2 months ago

This issue was closed because it has been stalled for 5 days with no activity.