search

falcosecurity / driverkit

Kit for building Falco drivers: kernel modules or eBPF probes
Apache License 2.0
64 stars 53 forks source link

Driverkit unable to build on Flatcar Stable (Kernel 5.10.25) #97

Closed jkbuster-work19 closed 2 years ago

jkbuster-work19 commented 3 years ago

Describe the bug

driverkit scripts do not work for current Flatcar release (Kernel 5.10.25)

How to reproduce it

configs/flatcar-2765.2.2.yaml:

kernelrelease: 5.10.25-flatcar
kernelversion: 1
target: vanilla
output:
  probe: /output/falco-flatcar-2765.2.2.o
driverversion: 2aa88dcf6243982697811df4c1b484bcbe9488a2

Run:

driverkit docker -c configs/flatcar-2765.2.2.yaml --timeout 300

Expected behaviour

eBPF probe builds for desired kernel

Screenshots

Initially, it appears that python3 is needed:

DEBU Auto-detecting system features:              
DEBU 3...                        libelf: [ on  ] 
DEBU 3...                          zlib: [ on  ] 
DEBU 4...                           bpf: [ on  ] 
DEBU                                              
DEBU B  GEN      /tmp/kernel/tools/bpf/resolve_btfids/bpf_helper_defs.h 
DEBU 3/usr/bin/env: 'python3': No such file or directory 
DEBU make[3]: *** [Makefile:182: /tmp/kernel/tools/bpf/resolve_btfids/bpf_helper_defs.h] Error 127 
DEBU make[3]: *** Deleting file '/tmp/kernel/tools/bpf/resolve_btfids/bpf_helper_defs.h' 
DEBU Smake[2]: *** [Makefile:47: /tmp/kernel/tools/bpf/resolve_btfids//libbpf.a] Error 2 
DEBU 7make[1]: *** [Makefile:71: bpf/resolve_btfids] Error 2 
DEBU <make: *** [Makefile:1933: tools/bpf/resolve_btfids] Error 2

After adding that into the driverkit-builder image, things get stuck due to -fmacro-prefix-map:

DEBU n+ cd /tmp/driver/bpf                 
DEBU + make LLC=/usr/bin/llc-7 CLANG=/usr/bin/clang-7 CC=/usr/bin/gcc-8 KERNELDIR=/tmp/kernel 
DEBU ake -C /tmp/kernel M=$PWD           
DEBU *make[1]: Entering directory '/tmp/kernel' 
DEBU Nscripts/Makefile.lib:8: 'always' is deprecated. Please use 'always-y' instead 
DEBU /usr/bin/clang-7 -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h \ 
DEBU    -D__KERNEL__ -fmacro-prefix-map=./=  \      
DEBU     \                                          
DEBU     \                                          
DEBU    -D__KERNEL__ \                              
DEBU    -D__BPF_TRACING__ \                         
DEBU    -Wno-gnu-variable-sized-type-not-at-end \   
DEBU    -Wno-address-of-packed-member \             
DEBU    -fno-jump-tables \                          
DEBU    -fno-stack-protector \                      
DEBU    -Wno-tautological-compare \                 
DEBU    -O2 -g -emit-llvm -c /tmp/driver/bpf/probe.c -o /tmp/driver/bpf/probe.ll 
DEBU clang: 2error: unknown argument: '-fmacro-prefix-map=./=' 
DEBU Lmake[2]: *** [/tmp/driver/bpf/Makefile:33: /tmp/driver/bpf/probe.o] Error 1 
DEBU 6make[1]: *** [Makefile:1807: /tmp/driver/bpf] Error 2 
DEBU )make[1]: Leaving directory '/tmp/kernel' 
DEBU %make: *** [Makefile:18: all] Error 2

Additional context

It appears that -fmacro-prefix-map wasn't added until clang-10 link, which is unavailable in the current Debian-stable release.

However, rebuilding the driverkit-builder image on Sid also throws errors when trying to run gcc-8, which is only available on Debian-stable:

DEBU n+ cd /tmp/driver/bpf                 
DEBU + make LLC=/usr/bin/llc-7 CLANG=/usr/bin/clang-7 CC=/usr/bin/gcc-8 KERNELDIR=/tmp/kernel 
DEBU ake -C /tmp/kernel M=$PWD           
DEBU *make[1]: Entering directory '/tmp/kernel' 
DEBU 3make[1]: /usr/bin/gcc-8: No such file or directory 
DEBU Nscripts/Makefile.lib:8: 'always' is deprecated. Please use 'always-y' instead 
DEBU /usr/bin/clang-7 -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h \ 
DEBU    -D__KERNEL__ \                              
DEBU     \                                          
DEBU     \                                          
DEBU    -D__KERNEL__ \                              
DEBU    -D__BPF_TRACING__ \                         
DEBU    -Wno-gnu-variable-sized-type-not-at-end \   
DEBU    -Wno-address-of-packed-member \             
DEBU    -fno-jump-tables \                          
DEBU    -fno-stack-protector \                      
DEBU    -Wno-tautological-compare \                 
DEBU    -O2 -g -emit-llvm -c /tmp/driver/bpf/probe.c -o /tmp/driver/bpf/probe.ll 
DEBU make[2]: /usr/bin/clang-7: No such file or directory 
DEBU make[2]: *** [/tmp/driver/bpf/Makefile:33: /tmp/driver/bpf/probe.o] Error 127 
DEBU 6make[1]: *** [Makefile:1807: /tmp/driver/bpf] Error 2 
DEBU )make[1]: Leaving directory '/tmp/kernel' 
DEBU %make: *** [Makefile:18: all] Error 2
muffin87 commented 3 years ago

I'm encountering the same issue.

Here is my config:

kernelrelease: 5.10.15-1
kernelversion: 1
target: centos
output:
  module: /tmp/falco-centos7.ko
  probe: /tmp/falco-probe-bpf.o
driverversion: 5c0b863ddade7a45568c0ac97d037422c9efb750

Here is the error message:

DEBU %+ ln -sf /usr/bin/gcc-8 /usr/bin/gcc
DEBU .+ cd /tmp/driver
DEBU + make KERNELDIR=/tmp/kernel
DEBU *make -C /tmp/kernel M=/tmp/driver modules
DEBU *make[1]: Entering directory '/tmp/kernel'
DEBU   CC [M]  /tmp/driver/main.o
DEBU -  CC [M]  /tmp/driver/dynamic_params_table.o
DEBU &  CC [M]  /tmp/driver/fillers_table.o
DEBU $  CC [M]  /tmp/driver/flags_table.o
DEBU #  CC [M]  /tmp/driver/ppm_events.o
DEBU $  CC [M]  /tmp/driver/ppm_fillers.o
DEBU $  CC [M]  /tmp/driver/event_table.o
DEBU &  CC [M]  /tmp/driver/syscall_table.o
DEBU $  CC [M]  /tmp/driver/ppm_cputime.o
DEBU   LD [M]  /tmp/driver/falco.o
DEBU %  MODPOST /tmp/driver/Module.symvers
DEBU "  CC [M]  /tmp/driver/falco.mod.o
DEBU   LD [M]  /tmp/driver/falco.ko
DEBU )make[1]: Leaving directory '/tmp/kernel'
DEBU + modinfo falco.ko
DEBU filename:       /tmp/driver/falco.ko
DEBU author:         sysdig inc
DEBU license:        GPL
DEBU srcversion:     B4FFE3CC859ECD9E48D3EB7
DEBU depends:
DEBU retpoline:      Y
DEBU name:           falco
DEBU vermagic:       5.10.15-1.el7.elrepo.x86_64 SMP mod_unload modversions
DEBU parm:           max_consumers:Maximum number of consumers that can simultaneously open the devices (uint)
DEBU parm:           verbose:Enable verbose logging (bool)
DEBU l+ cd /tmp/driver/bpf
DEBU + make LLC=/usr/bin/llc-7 CLANG=/usr/bin/clang-7 CC=/usr/bin/gcc KERNELDIR=/tmp/kernel
DEBU ake -C /tmp/kernel M=$PWD
DEBU *make[1]: Entering directory '/tmp/kernel'
DEBU Nscripts/Makefile.lib:8: 'always' is deprecated. Please use 'always-y' instead
DEBU /usr/bin/clang-7 -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h \
DEBU    -D__KERNEL__ -fmacro-prefix-map=./=  \
DEBU     \
DEBU     \
DEBU    -D__KERNEL__ \
DEBU    -D__BPF_TRACING__ \
DEBU    -Wno-gnu-variable-sized-type-not-at-end \
DEBU    -Wno-address-of-packed-member \
DEBU    -fno-jump-tables \
DEBU    -fno-stack-protector \
DEBU    -Wno-tautological-compare \
DEBU    -O2 -g -emit-llvm -c /tmp/driver/bpf/probe.c -o /tmp/driver/bpf/probe.ll
DEBU 9clang: error: unknown argument: '-fmacro-prefix-map=./='
DEBU Lmake[2]: *** [/tmp/driver/bpf/Makefile:33: /tmp/driver/bpf/probe.o] Error 1
DEBU 6make[1]: *** [Makefile:1801: /tmp/driver/bpf] Error 2
DEBU )make[1]: Leaving directory '/tmp/kernel'
DEBU %make: *** [Makefile:18: all] Error 2
goshlanguage commented 3 years ago

I tried to compile with vanilla upstream headers. I didn't run into the issues above but wanted to share my experience in case it's helpful to anyone.

I used the following manifest to try to compile a module and probe for v2605.6.0:

kernelrelease: 5.4.67
kernelversion: 1
target: vanilla
output:
  probe: /tmp/falco-flatcar-2605.6.0.o
  module: /tmp/falco-flatcar-2605.6.0.ko
driverversion: 2aa88dcf6243982697811df4c1b484bcbe9488a2
kernelconfigdata: <redacted>

Invoked with driverkit docker -c flatcar.yaml --timeout 300

This built successfully, but after rsyncing the built module to my test vagrant and running insmod, I am getting:

localhost core # insmod /home/core/falco-flatcar-2605.6.0.ko
insmod: ERROR: could not insert module /home/core/falco-flatcar-2605.6.0.ko: Invalid module format

I suspect that this is because flatcar maintains their own fork of the kernel that differs from vanilla.

For this reason, I think you would need to build a custom builder, eluded to in the README here:

You probably came here because you want to tell the Falco Drivers Build Grid to build drivers for a specific distro you care about.

If that distribution is not supported by driverkit, the Falco Drivers Build Grid will not be able to just build it as it does for other distros.

....

I looked at other builders to see what would be necessary to create one for flatcar, and it seems they all fetch kernel headers from URLs, such as in the case of ubuntu, and even vanilla.

The blocker I am facing currently is that I don't think Kinvolk publishes their headers files anywhere, but instead keep them on disk after an installation, mentioned here.

I tried searching their upstream for kernel images and such but headers aren't included: https://stable.release.flatcar-linux.net/

poiana commented 3 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

poiana commented 2 years ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

poiana commented 2 years ago

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community. /close

poiana commented 2 years ago

@poiana: Closing this issue.

In response to [this](https://github.com/falcosecurity/driverkit/issues/97#issuecomment-997356097): >Rotten issues close after 30d of inactivity. > >Reopen the issue with `/reopen`. > >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Provide feedback via https://github.com/falcosecurity/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.