Facing issue while trying to build ebpf on power Rhel 8

[ashwinik30]

Hi ,

Description : Trying to build ebpf on power Rhel8 machine

Procedure: git clone --recursive https://github.com/stackrox/falcosecurity-libs cd falcosecurity-libs/

mkdir build && cd build cmake -DUSE_BUNDLED_DEPS=true -DCREATE_TEST_TARGETS=OFF ../ make sinsp clang -v make driver cmake -DBUILD_BPF=true ../ make bpf

Linux kernel version : 4.18.0-372.9.1.el8.ppc64le

Error facing :

#define __lwsync()      __asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory")
        ^
<built-in>:310:9: note: previous definition is here
#define __lwsync __builtin_ppc_lwsync
        ^
1 warning generated.
error: <unknown>:0:0: in function bpf_sys_open_by_handle_at_x i32 (i8*): Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

error: <unknown>:0:0: in function bpf_sys_open_by_handle_at_x i32 (i8*): Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

error: <unknown>:0:0: in function bpf_sys_open_by_handle_at_x i32 (i8*): Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

error: <unknown>:0:0: in function bpf_sys_open_by_handle_at_x i32 (i8*): Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

error: <unknown>:0:0: in function bpf_sys_open_by_handle_at_x i32 (i8*): Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

But if I try the same on Rhel9.2 It works fine

[Molter73]

Hi @ashwinik30,

The compiler is telling you that this probe is using too many variables in the stack, you'll need to figure out if you can remove some of those. Since we don't use the filler, another possibility would be to exclude it by surrounding it with #ifndef __ppc64le__, but this is not something I'd like.

I don't have access to a power VM so I can't really help much more than this, sorry.

[Molter73]

Looks like that probe was split upstream not so long ago, maybe due to an error similar to this: https://github.com/falcosecurity/libs/pull/1300

Can you try compiling the probe using the branch from this PR? https://github.com/stackrox/collector/pull/1343 Remember to run git submodule update after checking out the branch so the falcosecurity-libs submodule is updated too.

[ashwinik30]

Hi @Molter73 Yes , I will try above

[ashwinik30]

Also Just in case as you said we dont use filler https://github.com/stackrox/collector/issues/1391#issuecomment-1779236341 I just did https://github.com/stackrox/falcosecurity-libs/blob/cef440c71467485027a0a636c99cc891f4b7b35b/driver/bpf/fillers.h#L3442 here instead true => false the error went off

output when i did it to false

[root@ibm-p9z-23-lp2 build]# make bpf
In file included from /root/falcosecurity-libs/driver/bpf/probe.c:16:
In file included from ./include/linux/sched.h:12:
In file included from ./arch/powerpc/include/asm/current.h:16:
In file included from ./arch/powerpc/include/asm/paca.h:19:
In file included from ./include/linux/rh_kabi.h:30:
In file included from ./include/linux/compiler.h:324:
./arch/powerpc/include/asm/barrier.h:49:9: warning: '__lwsync' macro redefined [-Wmacro-redefined]
#define __lwsync()      __asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory")
        ^
<built-in>:310:9: note: previous definition is here
#define __lwsync __builtin_ppc_lwsync
        ^
1 warning generated.
Built target bpf
[root@ibm-p9z-23-lp2 build]#

[root@ibm-p9z-23-lp2 falcosecurity-libs]# ls -l driver/bpf/probe.o
-rw-r--r--. 1 root root 1869768 Oct 25 09:12 driver/bpf/probe.o
[root@ibm-p9z-23-lp2 falcosecurity-libs]#
[root@ibm-p9z-23-lp2 falcosecurity-libs]#

fyi

[Molter73]

Weird, that true is not for enabling or disabling the filler, but rather it marks if it should be treated as a syscall or not. AFAICT, all it does is assign a pointer, that shouldn't change how the compiler analyzes stack depth: https://github.com/stackrox/falcosecurity-libs/blob/cef440c71467485027a0a636c99cc891f4b7b35b/driver/bpf/plumbing_helpers.h#L436-L442

[ashwinik30]

yes It is but that I have tried and worked ... and binary got generated on path I hope this generated binary gives us green signal of successfully builted ?

[ashwinik30]

will try this too and come back https://github.com/stackrox/collector/issues/1391#issuecomment-1779246311

[ashwinik30]

Hi @mauro As we have discussed comment I have tried building ebpf build on older branch mauro/use-falco-upstream

I am able to build ebpf successfully

[root@ibm-p9z-27-lp23 collector]# make -C kernel-modules drivers
make: Entering directory '/root/collector/kernel-modules'
docker build  -t build-kernel-modules-rhel8 ./build -f build/rhel8.Dockerfile
[+] Building 1.1s (9/9) FINISHED                                                                                                              docker:default
 => [internal] load build definition from rhel8.Dockerfile                                                                                              0.2s
 => => transferring dockerfile: 769B                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                       0.3s
 => => transferring context: 2B                                                                                                                         0.0s
 => [internal] load metadata for quay.io/centos/centos:stream8                                                                                          0.3s
 => [1/4] FROM quay.io/centos/centos:stream8@sha256:b1f6889548eda34b2ddc8c2f50a49bf9924164814308e41e90a07e3b30e0db7f                                    0.0s
 => [internal] load build context                                                                                                                       0.1s
 => => transferring context: 187B                                                                                                                       0.0s
 => CACHED [2/4] RUN dnf -y update &&     dnf -y install --nobest         make         cmake         gcc-c++         llvm         clang         gettex  0.0s
 => CACHED [3/4] COPY /build-kos /scripts/                                                                                                              0.0s
 => CACHED [4/4] COPY /build-wrapper.sh /scripts/compile.sh                                                                                             0.0s
 => exporting to image                                                                                                                                  0.0s
 => => exporting layers                                                                                                                                 0.0s
 => => writing image sha256:dbbf11acaf429e683b6f1e235c06422e1e47d19836666269c52959f028401c54                                                            0.0s
 => => naming to docker.io/library/build-kernel-modules-rhel8                                                                                           0.0s
docker run --rm \
        --entrypoint "/bin/bash" \
        -v /root/collector/kernel-modules/..:/collector \
        -v /lib/modules/:/lib/modules/:ro \
        -v /usr/src:/usr/src:ro \
        -e GIT_COMMIT=3eb4593f495749f7f6f658afcae79a187806c40a \
        build-kernel-modules-rhel8:latest \
        -c /collector/kernel-modules/dev/build-drivers.sh
make: Entering directory '/collector/kernel-modules/probe'
make -C /lib/modules/4.18.0-477.10.1.el8_8.ppc64le/build M=$PWD
make[1]: Entering directory '/usr/src/kernels/4.18.0-477.10.1.el8_8.ppc64le'
clang -I./arch/powerpc/include -I./arch/powerpc/include/generated   -I./include/drm-backport -I./include -I./arch/powerpc/include/uapi -I./arch/powerpc/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h \
        -D__KERNEL__ -Iarch/powerpc -DHAVE_AS_ATHIGH=1   -DBPF_SKIP_CPUSETS -DBPF_SKIP_CPUACCT -DBPF_SKIP_BLK_CGROUP \
         \
         \
        -I /collector/falcosecurity-libs/driver/bpf \
        -D__KERNEL__ \
        -DKBUILD_MODNAME=\"collector\" \
        -D__BPF_TRACING__ \
        -Wno-gnu-variable-sized-type-not-at-end \
        -Wno-address-of-packed-member \
        -fno-jump-tables \
        -fno-stack-protector \
        -Wno-tautological-compare \
        -O2  -emit-llvm -c /collector/kernel-modules/probe/collector_probe.c -o /collector/kernel-modules/probe/probe.ll
In file included from /collector/kernel-modules/probe/collector_probe.c:16:
In file included from ./include/linux/sched.h:12:
In file included from ./arch/powerpc/include/asm/current.h:16:
In file included from ./arch/powerpc/include/asm/paca.h:19:
In file included from ./include/linux/rh_kabi.h:30:
In file included from ./include/linux/compiler.h:326:
./arch/powerpc/include/asm/barrier.h:49:9: warning: '__lwsync' macro redefined [-Wmacro-redefined]
#define __lwsync()      __asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory")
        ^
<built-in>:323:9: note: previous definition is here
#define __lwsync __builtin_ppc_lwsync
        ^
1 warning generated.
llc -march=bpf -filetype=obj -o /collector/kernel-modules/probe/probe.o /collector/kernel-modules/probe/probe.ll
  Building modules, stage 2.
  MODPOST 0 modules
make[1]: Leaving directory '/usr/src/kernels/4.18.0-477.10.1.el8_8.ppc64le'
make: Leaving directory '/collector/kernel-modules/probe'
make: Leaving directory '/root/collector/kernel-modules'

[root@ibm-p9z-27-lp23 collector]# git status
On branch mauro/use-falco-upstream
Your branch is up to date with 'origin/mauro/use-falco-upstream'

Thanks :)

[ashwinik30]

[root@ibm-p9z-27-lp23 collector]# cd kernel-modules/probe
[root@ibm-p9z-27-lp23 probe]# ls
collector_probe.c  collector_probe.h  Makefile  modules.order  Module.symvers  probe.ll  probe.o

[ashwinik30]

^^ @Molter73 @pratham-m