Closed Jeroen0494 closed 1 year ago
FedeDP
commented
2 years ago Hi! Thanks for reporting this issue! We are aware of this, see eg: #126 .
Good news is: we are working on a feature(ie: supporting multiple builder images) that will fix this issue in #202!
FedeDP
commented
2 years ago Hi! Can you retest using driverkit master and forcing --builderimage auto:master ?
Jeroen0494
commented
2 years ago Hi,
Back from vacation. So that seems to work better:
+ rm -Rf /tmp/driver
+ mkdir /tmp/driver
+ rm -Rf /tmp/module-download
+ mkdir -p /tmp/module-download
+ tar -xzf - -C /tmp/module-download
+ curl --silent -SL https://github.com/falcosecurity/libs/archive/master.tar.gz
+ mv /tmp/module-download/libs-master/driver/API_VERSION /tmp/module-download/libs-master/driver/CMakeLists.txt /tmp/module-download/libs-master/driver/GPL2.txt /tmp/module-download/libs-master/driver/MIT.txt /tmp/module-download/libs-master/driver/Makefile.in /tmp/module-download/libs-master/driver/README.VERSION.md /tmp/module-download/libs-master/driver/SCHEMA_VERSION /tmp/module-download/libs-master/driver/bpf /tmp/module-download/libs-master/driver/dkms.conf.in /tmp/module-download/libs-master/driver/driver_config.h.in /tmp/module-download/libs-master/driver/dynamic_params_table.c /tmp/module-download/libs-master/driver/event_table.c /tmp/module-download/libs-master/driver/feature_gates.h /tmp/module-download/libs-master/driver/fillers_table.c /tmp/module-download/libs-master/driver/flags_table.c /tmp/module-download/libs-master/driver/kernel_hacks.h /tmp/module-download/libs-master/driver/main.c /tmp/module-download/libs-master/driver/modern_bpf /tmp/module-download/libs-master/driver/ppm.h /tmp/module-download/libs-master/driver/ppm_api_version.h /tmp/module-download/libs-master/driver/ppm_compat_unistd_32.h /tmp/module-download/libs-master/driver/ppm_cputime.c /tmp/module-download/libs-master/driver/ppm_events.c /tmp/module-download/libs-master/driver/ppm_events.h /tmp/module-download/libs-master/driver/ppm_events_public.h /tmp/module-download/libs-master/driver/ppm_fillers.c /tmp/module-download/libs-master/driver/ppm_fillers.h /tmp/module-download/libs-master/driver/ppm_flag_helpers.h /tmp/module-download/libs-master/driver/ppm_ringbuffer.h /tmp/module-download/libs-master/driver/ppm_syscall.h /tmp/module-download/libs-master/driver/ppm_tp.h /tmp/module-download/libs-master/driver/ppm_version.h /tmp/module-download/libs-master/driver/syscall_table.c /tmp/module-download/libs-master/driver/systype_compat.h /tmp/module-download/libs-master/driver/tp_table.c /tmp/driver
+ cp /driverkit/module-Makefile /tmp/driver/Makefile
+ bash /driverkit/fill-driver-config.sh /tmp/driver
+ DRIVER_BUILD_DIR=/tmp/driver
+ DRIVER_CONFIG_FILE=/tmp/driver/driver_config.h
+ cat
+ API_VERSION_FILE=/tmp/driver/API_VERSION
+ [[ -f /tmp/driver/API_VERSION ]]
++ cut -f 1 -d . /tmp/driver/API_VERSION
+ PPM_API_CURRENT_VERSION_MAJOR=2
++ cut -f 2 -d . /tmp/driver/API_VERSION
+ PPM_API_CURRENT_VERSION_MINOR=0
++ cut -f 3 -d . /tmp/driver/API_VERSION
+ PPM_API_CURRENT_VERSION_PATCH=0
+ echo '#define PPM_API_CURRENT_VERSION_MAJOR' 2
+ echo '#define PPM_API_CURRENT_VERSION_MINOR' 0
+ echo '#define PPM_API_CURRENT_VERSION_PATCH' 0
+ SCHEMA_VERSION_FILE=/tmp/driver/SCHEMA_VERSION
+ [[ -f /tmp/driver/SCHEMA_VERSION ]]
++ cut -f 1 -d . /tmp/driver/SCHEMA_VERSION
+ PPM_SCHEMA_CURRENT_VERSION_MAJOR=2
++ cut -f 2 -d . /tmp/driver/SCHEMA_VERSION
+ PPM_SCHEMA_CURRENT_VERSION_MINOR=1
++ cut -f 3 -d . /tmp/driver/SCHEMA_VERSION
+ PPM_SCHEMA_CURRENT_VERSION_PATCH=0
+ echo '#define PPM_SCHEMA_CURRENT_VERSION_MAJOR' 2
+ echo '#define PPM_SCHEMA_CURRENT_VERSION_MINOR' 1
+ echo '#define PPM_SCHEMA_CURRENT_VERSION_PATCH' 0
+ echo '#include "ppm_api_version.h"'
+ mkdir /tmp/kernel-download
+ cd /tmp/kernel-download
+ curl --silent -o kernel.deb -SL https://mirrors.edge.kernel.org/ubuntu/pool/main/l/linux/linux-headers-5.15.0-47-generic_5.15.0-47.51_amd64.deb
+ ar x kernel.deb
+ tar -xf data.tar.zst
+ curl --silent -o kernel.deb -SL https://mirrors.edge.kernel.org/ubuntu/pool/main/l/linux/linux-headers-5.15.0-47_5.15.0-47.51_all.deb
+ ar x kernel.deb
+ tar -xf data.tar.zst
+ cd /tmp/kernel-download/usr/src/
++ find . -type d -name 'linux-headers*generic'
++ head -n 1
++ xargs readlink -f
+ sourcedir=/tmp/kernel-download/usr/src/linux-headers-5.15.0-47-generic
+ cd /tmp/driver
+ make CC=/usr/bin/gcc-11 KERNELDIR=/tmp/kernel-download/usr/src/linux-headers-5.15.0-47-generic
make -C /tmp/kernel-download/usr/src/linux-headers-5.15.0-47-generic M=/tmp/driver modules
make[1]: Entering directory '/tmp/kernel-download/usr/src/linux-headers-5.15.0-47-generic'
warning: the compiler differs from the one used to build the kernel
The kernel was built by: gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
You are using: gcc-11 (Debian 11.3.0-5) 11.3.0
CC [M] /tmp/driver/main.o
CC [M] /tmp/driver/dynamic_params_table.o
CC [M] /tmp/driver/fillers_table.o
CC [M] /tmp/driver/flags_table.o
CC [M] /tmp/driver/ppm_events.o
CC [M] /tmp/driver/ppm_fillers.o
CC [M] /tmp/driver/event_table.o
CC [M] /tmp/driver/syscall_table.o
CC [M] /tmp/driver/ppm_cputime.o
CC [M] /tmp/driver/tp_table.o
LD [M] /tmp/driver/falco.o
MODPOST /tmp/driver/Module.symvers
CC [M] /tmp/driver/falco.mod.o
LD [M] /tmp/driver/falco.ko
Skipping BTF generation for /tmp/driver/falco.ko due to unavailability of vmlinux
BTF [M] /tmp/driver/falco.ko
make[1]: Leaving directory '/tmp/kernel-download/usr/src/linux-headers-5.15.0-47-generic'
+ mv falco.ko /tmp/driver/module.ko
+ strip -g /tmp/driver/module.ko
+ modinfo /tmp/driver/module.ko
filename: /tmp/driver/module.ko
schema_version: 2.1.0
api_version: 2.0.0
build_commit: master
version: master
author: the Falco authors
license: GPL
srcversion: 5EA976FE1BA6B761E4EDA56
depends:
retpoline: Y
name: falco
vermagic: 5.15.0-47-generic SMP mod_unload modversions
parm: g_buffer_bytes_dim:This is the dimension of a single per-CPU buffer in bytes. Please note: this buffer will be mapped twice in the process virtual memory, so pay attention to its size.
parm: max_consumers:Maximum number of consumers that can simultaneously open the devices (uint)
parm: verbose:Enable verbose logging (bool)
+ cd /tmp/driver/bpf
+ make KERNELDIR=/tmp/kernel-download/usr/src/linux-headers-5.15.0-47-generic
make -C /tmp/kernel-download/usr/src/linux-headers-5.15.0-47-generic M=$PWD
make[1]: Entering directory '/tmp/kernel-download/usr/src/linux-headers-5.15.0-47-generic'
warning: the compiler differs from the one used to build the kernel
The kernel was built by: gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
You are using: gcc (Debian 12.2.0-3) 12.2.0
clang -I./arch/x86/include -I./arch/x86/include/generated -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -I./ubuntu/include \
-D__KERNEL__ -fmacro-prefix-map=./= \
\
\
-D__KERNEL__ \
-D__BPF_TRACING__ \
-Wno-gnu-variable-sized-type-not-at-end \
-Wno-address-of-packed-member \
-fno-jump-tables \
-fno-stack-protector \
-Wno-tautological-compare \
-O2 -g -emit-llvm -c /tmp/driver/bpf/probe.c -o /tmp/driver/bpf/probe.ll
llc -march=bpf -filetype=obj -o /tmp/driver/bpf/probe.o /tmp/driver/bpf/probe.ll
MODPOST /tmp/driver/bpf/Module.symvers
make[1]: Leaving directory '/tmp/kernel-download/usr/src/linux-headers-5.15.0-47-generic'
+ ls -l probe.o
-rw-r--r-- 1 root root 3991624 Sep 30 13:10 probe.o
+ rm /tmp/module.lock
+ rm /tmp/probe.lock
+ touch /tmp/download.lock
+ true
+ '[' -f /tmp/download.lock ']'
+ echo 'Lock not released yet - waiting for 5 seconds'
+ sleep 5
Lock not released yet - waiting for 5 seconds
download lock was released, we can exit now
+ continue
+ true
+ '[' -f /tmp/download.lock ']'
+ echo 'download lock was released, we can exit now'
+ break
rpc error: code = NotFound desc = an error occurred when try to find container "828080744e0c21b2ab7abaa4ad2d5589bb8f4e417d340fc7cf8a2183c9fc25ec": not foundI'll try and use it for my cluster a bit later, just got back today.
FedeDP
commented
2 years ago Yay! Let us know please, this is a super useful feedback for us!
Jeroen0494
commented
2 years ago Hi,
So I haven't gotten it to work just yet with BPF:
$ kubectl logs -n falco falco-2s9n5
Mon Oct 10 18:24:13 2022: Falco version 0.32.2
Mon Oct 10 18:24:13 2022: Falco initialized with configuration file /etc/falco/falco.yaml
Mon Oct 10 18:24:13 2022: Loading rules from file /etc/falco/falco_rules.yaml:
Mon Oct 10 18:24:14 2022: Loading rules from file /etc/falco/falco_rules.local.yaml:
Mon Oct 10 18:24:14 2022: Starting internal webserver, listening on port 8765
Mon Oct 10 18:24:14 2022: Unable to load the driver.
Mon Oct 10 18:24:14 2022: Runtime error: can't create map: Errno 22. Exiting.I've saved the falco-bpf.o file under /root/.falco on my host and mapped it into the container using hostPath.
Using the regular module doesn't seem to work yet either:
$ ls -l /dev/falco0
cr-------- 1 root root 509, 0 okt 10 18:26 /dev/falco0
jeroen@mediaserver:~/Kubernetes/k3s/falco$ k get pod -n falco
NAME READY STATUS RESTARTS AGE
falco-nqjqp 0/1 CrashLoopBackOff 2 (12s ago) 43s
jeroen@mediaserver:~/Kubernetes/k3s/falco$ k logs -n falco falco-nqjqp
Mon Oct 10 18:31:10 2022: Falco version 0.32.2
Mon Oct 10 18:31:10 2022: Falco initialized with configuration file /etc/falco/falco.yaml
Mon Oct 10 18:31:10 2022: Loading rules from file /etc/falco/falco_rules.yaml:
Mon Oct 10 18:31:10 2022: Loading rules from file /etc/falco/falco_rules.local.yaml:
Mon Oct 10 18:31:10 2022: Starting internal webserver, listening on port 8765
Mon Oct 10 18:31:10 2022: Unable to load the driver.
Mon Oct 10 18:31:10 2022: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.But I'm guessing in these cases I'm doing something wrong. I inserted the falco module using insmod. Falco is deployed via Helm.
Jeroen0494
commented
2 years ago Using the master image I get the following:
BPF
$ kubectl logs -n falco falco-lwl4t
Mon Oct 10 18:35:15 2022: Falco version: 0.32.1-241+79d875c (x86_64)
Mon Oct 10 18:35:15 2022: Falco initialized with configuration file: /etc/falco/falco.yaml
Mon Oct 10 18:35:15 2022: Loading rules from file /etc/falco/falco_rules.yaml
Mon Oct 10 18:35:15 2022: Loading rules from file /etc/falco/falco_rules.local.yaml
Mon Oct 10 18:35:15 2022: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Mon Oct 10 18:35:15 2022: Starting health webserver with threadiness 4, listening on port 8765
Mon Oct 10 18:35:15 2022: Enabled event sources: syscall
Mon Oct 10 18:35:15 2022: Opening capture with BPF probe. BPF probe path: /root/.falco/falco-bpf.o
Error: pmu_fd < 0: Errno 13
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:Kernel module:
$ k logs -n falco falco-sn9zw
Mon Oct 10 18:37:20 2022: Falco version: 0.32.1-241+79d875c (x86_64)
Mon Oct 10 18:37:20 2022: Falco initialized with configuration file: /etc/falco/falco.yaml
Mon Oct 10 18:37:20 2022: Loading rules from file /etc/falco/falco_rules.yaml
Mon Oct 10 18:37:20 2022: Loading rules from file /etc/falco/falco_rules.local.yaml
Mon Oct 10 18:37:20 2022: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Mon Oct 10 18:37:20 2022: Starting health webserver with threadiness 4, listening on port 8765
Mon Oct 10 18:37:20 2022: Enabled event sources: syscall
Mon Oct 10 18:37:20 2022: Opening capture with Kernel module
Mon Oct 10 18:37:20 2022: Trying to inject the Kernel module and opening the capture again...
Mon Oct 10 18:37:20 2022: Unable to load the driver
Error: unable to open '/sys/module/falco/parameters/g_buffer_bytes_dim': Errno 30. Please ensure the kernel module is already loaded.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:BPF
Yay!
Kernel module:
We are currently fixing it for the 0.33 Falco release :)
Jeroen0494
commented
2 years ago Hi,
It fails to build again when building against the latest kernel (5.15.0-50-generic):
kernelrelease: 5.15.0-50-generic
kernelversion: 56
target: ubuntu-generic
output:
module: /tmp/falco.ko
probe: /tmp/falco.o
driverversion: master
builderimage: falcosecurity/driverkit-builder:v0.9.7+ rm -Rf /tmp/driver
+ mkdir /tmp/driver
+ rm -Rf /tmp/module-download
+ mkdir -p /tmp/module-download
+ curl --silent -SL https://github.com/falcosecurity/libs/archive/master.tar.gz
+ tar -xzf - -C /tmp/module-download
+ mv /tmp/module-download/libs-master/driver/API_VERSION /tmp/module-download/libs-master/driver/CMakeLists.txt /tmp/module-download/libs-master/driver/GPL2.txt /tmp/module-download/libs-master/driver/MIT.txt /tmp/module-download/libs-master/driver/Makefile.in /tmp/module-download/libs-master/driver/README.VERSION.md /tmp/module-download/libs-master/driver/SCHEMA_VERSION /tmp/module-download/libs-master/driver/bpf /tmp/module-download/libs-master/driver/dkms.conf.in /tmp/module-download/libs-master/driver/driver_config.h.in /tmp/module-download/libs-master/driver/dynamic_params_table.c /tmp/module-download/libs-master/driver/event_table.c /tmp/module-download/libs-master/driver/feature_gates.h /tmp/module-download/libs-master/driver/fillers_table.c /tmp/module-download/libs-master/driver/flags_table.c /tmp/module-download/libs-master/driver/kernel_hacks.h /tmp/module-download/libs-master/driver/main.c /tmp/module-download/libs-master/driver/modern_bpf /tmp/module-download/libs-master/driver/ppm.h /tmp/module-download/libs-master/driver/ppm_api_version.h /tmp/module-download/libs-master/driver/ppm_compat_unistd_32.h /tmp/module-download/libs-master/driver/ppm_cputime.c /tmp/module-download/libs-master/driver/ppm_events.c /tmp/module-download/libs-master/driver/ppm_events.h /tmp/module-download/libs-master/driver/ppm_events_public.h /tmp/module-download/libs-master/driver/ppm_fillers.c /tmp/module-download/libs-master/driver/ppm_fillers.h /tmp/module-download/libs-master/driver/ppm_flag_helpers.h /tmp/module-download/libs-master/driver/ppm_ringbuffer.h /tmp/module-download/libs-master/driver/ppm_syscall.h /tmp/module-download/libs-master/driver/ppm_tp.h /tmp/module-download/libs-master/driver/ppm_version.h /tmp/module-download/libs-master/driver/syscall_compat_aarch64.h /tmp/module-download/libs-master/driver/syscall_compat_s390x.h /tmp/module-download/libs-master/driver/syscall_compat_x86_64.h /tmp/module-download/libs-master/driver/syscall_table.c /tmp/module-download/libs-master/driver/systype_compat.h /tmp/module-download/libs-master/driver/tp_table.c /tmp/driver
+ cp /driverkit/module-Makefile /tmp/driver/Makefile
+ bash /driverkit/fill-driver-config.sh /tmp/driver
+ DRIVER_BUILD_DIR=/tmp/driver
+ DRIVER_CONFIG_FILE=/tmp/driver/driver_config.h
+ cat
+ API_VERSION_FILE=/tmp/driver/API_VERSION
+ [[ -f /tmp/driver/API_VERSION ]]
++ cut -f 1 -d . /tmp/driver/API_VERSION
+ PPM_API_CURRENT_VERSION_MAJOR=2
++ cut -f 2 -d . /tmp/driver/API_VERSION
+ PPM_API_CURRENT_VERSION_MINOR=0
++ cut -f 3 -d . /tmp/driver/API_VERSION
+ PPM_API_CURRENT_VERSION_PATCH=0
+ echo '#define PPM_API_CURRENT_VERSION_MAJOR' 2
+ echo '#define PPM_API_CURRENT_VERSION_MINOR' 0
+ echo '#define PPM_API_CURRENT_VERSION_PATCH' 0
+ SCHEMA_VERSION_FILE=/tmp/driver/SCHEMA_VERSION
+ [[ -f /tmp/driver/SCHEMA_VERSION ]]
++ cut -f 1 -d . /tmp/driver/SCHEMA_VERSION
+ PPM_SCHEMA_CURRENT_VERSION_MAJOR=2
++ cut -f 2 -d . /tmp/driver/SCHEMA_VERSION
+ PPM_SCHEMA_CURRENT_VERSION_MINOR=1
++ cut -f 3 -d . /tmp/driver/SCHEMA_VERSION
+ PPM_SCHEMA_CURRENT_VERSION_PATCH=0
+ echo '#define PPM_SCHEMA_CURRENT_VERSION_MAJOR' 2
+ echo '#define PPM_SCHEMA_CURRENT_VERSION_MINOR' 1
+ echo '#define PPM_SCHEMA_CURRENT_VERSION_PATCH' 0
+ echo '#include "ppm_api_version.h"'
+ mkdir /tmp/kernel-download
+ cd /tmp/kernel-download
+ curl --silent -o kernel.deb -SL https://mirrors.edge.kernel.org/ubuntu/pool/main/l/linux/linux-headers-5.15.0-50-generic_5.15.0-50.56_amd64.deb
+ ar x kernel.deb
+ tar -xf data.tar.zst
+ curl --silent -o kernel.deb -SL https://mirrors.edge.kernel.org/ubuntu/pool/main/l/linux/linux-headers-5.15.0-50_5.15.0-50.56_all.deb
+ ar x kernel.deb
+ tar -xf data.tar.zst
+ cd /tmp/kernel-download/usr/src/
++ xargs readlink -f
++ head -n 1
++ find . -type d -name 'linux-headers*generic'
+ sourcedir=/tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic
+ cd /tmp/driver
+ make CC=/usr/bin/gcc-10 KERNELDIR=/tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic
make -C /tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic M=/tmp/driver modules
make[1]: Entering directory '/tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic'
arch/x86/Makefile:142: CONFIG_X86_X32 enabled but no binutils support
make[1]: /usr/bin/gcc-10: Command not found
warning: the compiler differs from the one used to build the kernel
The kernel was built by: gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
You are using:
CC [M] /tmp/driver/main.o
/bin/sh: 1: /usr/bin/gcc-10: not found
make[2]: *** [scripts/Makefile.build:297: /tmp/driver/main.o] Error 127
make[1]: *** [Makefile:1900: /tmp/driver] Error 2
make[1]: Leaving directory '/tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic'
make: *** [Makefile:7: all] Error 2I've attempted to use driver-builder_bullseye because it has a 0.10.1 release, but this clearly doesn't work:
+ rm -Rf /tmp/driver
+ mkdir /tmp/driver
+ rm -Rf /tmp/module-download
+ mkdir -p /tmp/module-download
+ curl --silent -SL https://github.com/falcosecurity/libs/archive/master.tar.gz
+ tar -xzf - -C /tmp/module-download
+ mv /tmp/module-download/libs-master/driver/API_VERSION /tmp/module-download/libs-master/driver/CMakeLists.txt /tmp/module-download/libs-master/driver/GPL2.txt /tmp/module-download/libs-master/driver/MIT.txt /tmp/module-download/libs-master/driver/Makefile.in /tmp/module-download/libs-master/driver/README.VERSION.md /tmp/module-download/libs-master/driver/SCHEMA_VERSION /tmp/module-download/libs-master/driver/bpf /tmp/module-download/libs-master/driver/dkms.conf.in /tmp/module-download/libs-master/driver/driver_config.h.in /tmp/module-download/libs-master/driver/dynamic_params_table.c /tmp/module-download/libs-master/driver/event_table.c /tmp/module-download/libs-master/driver/feature_gates.h /tmp/module-download/libs-master/driver/fillers_table.c /tmp/module-download/libs-master/driver/flags_table.c /tmp/module-download/libs-master/driver/kernel_hacks.h /tmp/module-download/libs-master/driver/main.c /tmp/module-download/libs-master/driver/modern_bpf /tmp/module-download/libs-master/driver/ppm.h /tmp/module-download/libs-master/driver/ppm_api_version.h /tmp/module-download/libs-master/driver/ppm_compat_unistd_32.h /tmp/module-download/libs-master/driver/ppm_cputime.c /tmp/module-download/libs-master/driver/ppm_events.c /tmp/module-download/libs-master/driver/ppm_events.h /tmp/module-download/libs-master/driver/ppm_events_public.h /tmp/module-download/libs-master/driver/ppm_fillers.c /tmp/module-download/libs-master/driver/ppm_fillers.h /tmp/module-download/libs-master/driver/ppm_flag_helpers.h /tmp/module-download/libs-master/driver/ppm_ringbuffer.h /tmp/module-download/libs-master/driver/ppm_syscall.h /tmp/module-download/libs-master/driver/ppm_tp.h /tmp/module-download/libs-master/driver/ppm_version.h /tmp/module-download/libs-master/driver/syscall_compat_aarch64.h /tmp/module-download/libs-master/driver/syscall_compat_s390x.h /tmp/module-download/libs-master/driver/syscall_compat_x86_64.h /tmp/module-download/libs-master/driver/syscall_table.c /tmp/module-download/libs-master/driver/systype_compat.h /tmp/module-download/libs-master/driver/tp_table.c /tmp/driver
+ cp /driverkit/module-Makefile /tmp/driver/Makefile
+ bash /driverkit/fill-driver-config.sh /tmp/driver
+ DRIVER_BUILD_DIR=/tmp/driver
+ DRIVER_CONFIG_FILE=/tmp/driver/driver_config.h
+ cat
+ API_VERSION_FILE=/tmp/driver/API_VERSION
+ [[ -f /tmp/driver/API_VERSION ]]
++ cut -f 1 -d . /tmp/driver/API_VERSION
+ PPM_API_CURRENT_VERSION_MAJOR=2
++ cut -f 2 -d . /tmp/driver/API_VERSION
+ PPM_API_CURRENT_VERSION_MINOR=0
++ cut -f 3 -d . /tmp/driver/API_VERSION
+ PPM_API_CURRENT_VERSION_PATCH=0
+ echo '#define PPM_API_CURRENT_VERSION_MAJOR' 2
+ echo '#define PPM_API_CURRENT_VERSION_MINOR' 0
+ echo '#define PPM_API_CURRENT_VERSION_PATCH' 0
+ SCHEMA_VERSION_FILE=/tmp/driver/SCHEMA_VERSION
+ [[ -f /tmp/driver/SCHEMA_VERSION ]]
++ cut -f 1 -d . /tmp/driver/SCHEMA_VERSION
+ PPM_SCHEMA_CURRENT_VERSION_MAJOR=2
++ cut -f 2 -d . /tmp/driver/SCHEMA_VERSION
+ PPM_SCHEMA_CURRENT_VERSION_MINOR=1
++ cut -f 3 -d . /tmp/driver/SCHEMA_VERSION
+ PPM_SCHEMA_CURRENT_VERSION_PATCH=0
+ echo '#define PPM_SCHEMA_CURRENT_VERSION_MAJOR' 2
+ echo '#define PPM_SCHEMA_CURRENT_VERSION_MINOR' 1
+ echo '#define PPM_SCHEMA_CURRENT_VERSION_PATCH' 0
+ echo '#include "ppm_api_version.h"'
+ mkdir /tmp/kernel-download
+ cd /tmp/kernel-download
+ curl --silent -o kernel.deb -SL https://mirrors.edge.kernel.org/ubuntu/pool/main/l/linux/linux-headers-5.15.0-50-generic_5.15.0-50.56_amd64.deb
+ ar x kernel.deb
+ tar -xf data.tar.zst
+ curl --silent -o kernel.deb -SL https://mirrors.edge.kernel.org/ubuntu/pool/main/l/linux/linux-headers-5.15.0-50_5.15.0-50.56_all.deb
+ ar x kernel.deb
+ tar -xf data.tar.zst
+ cd /tmp/kernel-download/usr/src/
++ find . -type d -name 'linux-headers*generic'
++ head -n 1
++ xargs readlink -f
+ sourcedir=/tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic
+ cd /tmp/driver
+ make CC=/usr/bin/gcc-10 KERNELDIR=/tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic
make -C /tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic M=/tmp/driver modules
make[1]: Entering directory '/tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic'
warning: the compiler differs from the one used to build the kernel
The kernel was built by: gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
You are using: gcc-10 (Debian 10.2.1-6) 10.2.1 20210110
CC [M] /tmp/driver/main.o
gcc-10: error: unrecognized command-line option '-mharden-sls=all'
make[2]: *** [scripts/Makefile.build:297: /tmp/driver/main.o] Error 1
make[1]: *** [Makefile:1900: /tmp/driver] Error 2
make[1]: Leaving directory '/tmp/kernel-download/usr/src/linux-headers-5.15.0-50-generic'
make: *** [Makefile:7: all] Error 2Could you make the 0.10.1 image available on Docker hub for Ubuntu?
FedeDP
commented
2 years ago You are not using latest driverkit release. Can you use 0.10.1? It should allow you to build the driver for that kernel!
EDIT: when using latest driverkit, please do not even specify any builder image!
Jeroen0494
commented
2 years ago Ah, my bad. Build using the lastest version of driverkit, now it works. Running falco doesn't work just yet:
kubectl logs -n falco falco-mxxfz
Sat Oct 15 13:37:09 2022: Falco version 0.32.0 (driver version 39ae7d40496793cf3d3e7890c9bbdc202263836b)
Sat Oct 15 13:37:09 2022: Falco initialized with configuration file /etc/falco/falco.yaml
Sat Oct 15 13:37:09 2022: Loading rules from file /etc/falco/falco_rules.yaml:
Sat Oct 15 13:37:10 2022: Loading rules from file /etc/falco/falco_rules.local.yaml:
Sat Oct 15 13:37:10 2022: Starting internal webserver, listening on port 8765
Sat Oct 15 13:37:10 2022: Unable to load the driver.
Sat Oct 15 13:37:10 2022: Runtime error: can't create map: Errno 22. Exiting.Are you trying to run Falco with eBPF right? Btw we are nearing the release of Falco 0.33; i am 99% sure it will fix your issue ;)
Jeroen0494
commented
2 years ago Are you trying to run Falco with eBPF right? Btw we are nearing the release of Falco 0.33; i am 99% sure it will fix your issue ;)
Yes, because the kernel module doesn't seem to load. Alright, I'll wait for the next release.
poiana
commented
1 year ago Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
FedeDP
commented
1 year ago @Jeroen0494 had you the chance to upgrade to latest Falco? Did it work?
Jeroen0494
commented
1 year ago Hi,
I tried it again after your comment, it seems not to work yet. It seems to fetch the module from your repository instead of using the one I compiled myself.
jeroen@mediaserver:~/kubernetes/k3s/falco$ k logs -n falco falco-rmmx5 -c falcoctl-artifact-install
INFO: Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
WARN: No configured index. Consider to configure one using the 'index add' command.
INFO: Installing the following artifacts: [ghcr.io/falcosecurity/rules/falco-rules:0]
INFO: Preparing to pull "ghcr.io/falcosecurity/rules/falco-rules:0"
INFO: Retrieving credentials from local store
INFO: proceeding with empty credentials for registry "ghcr.io"
INFO: Pulling ad24f8acf278
INFO: Pulling 0d3705a4650f
INFO: Pulling 0957c1ef3fe4
INFO: Extracting and installing "rulesfile" "falco_rules.yaml.tar.gz"
INFO: Artifact successfully installed in "/rulesfiles"
jeroen@mediaserver:~/kubernetes/k3s/falco$ k logs -n falco falco-rmmx5 -c falco-driver-loader
* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.34.0, driver version=4.0.0+driver, arch=x86_64, kernel release=5.15.0-60-generic, kernel version=66
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
================ Cleaning phase ================
* 1. Check if kernel module 'falco' is still loaded:
- OK! There is no 'falco' module loaded.
* 2. Check all versions of kernel module 'falco' in dkms:
- OK! There are no 'falco' module versions in dkms.
[SUCCESS] Cleaning phase correctly terminated.
================ Cleaning phase ================
* Looking for a falco module locally (kernel 5.15.0-60-generic)
* Filename 'falco_ubuntu-generic_5.15.0-60-generic_66.ko' is composed of:
- driver name: falco
- target identifier: ubuntu-generic
- kernel release: 5.15.0-60-generic
- kernel version: 66
* Trying to download a prebuilt falco module from https://download.falco.org/driver/4.0.0%2Bdriver/x86_64/falco_ubuntu-generic_5.15.0-60-generic_66.ko
* Download succeeded
* Success: falco module found and inserted
jeroen@mediaserver:~/kubernetes/k3s/falco$ k logs -n falco falco-rmmx5 -c falco
Fri Feb 17 09:42:56 2023: Falco version: 0.34.0 (x86_64)
Fri Feb 17 09:42:56 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Feb 17 09:42:56 2023: Loading rules from file /etc/falco/falco_rules.yaml
Error: Could not create inotify handler
jeroen@mediaserver:~/kubernetes/k3s/falco$ k logs -n falco falco-rmmx5 -c falcoctl-artifact-follow
INFO: Retrieving versions from Falco (timeout 2m0s) ...
INFO: error: dial tcp [::1]:8765: connect: connection refused. Trying again in 1s
INFO: error: dial tcp [::1]:8765: connect: connection refused. Trying again in 1.6s
INFO: error: dial tcp [::1]:8765: connect: connection refused. Trying again in 2.56s
INFO: error: dial tcp [::1]:8765: connect: connection refused. Trying again in 4.096s
INFO: error: dial tcp [::1]:8765: connect: connection refused. Trying again in 6.5536s
INFO: error: dial tcp [::1]:8765: connect: connection refused. Trying again in 10.48576s
INFO: error: dial tcp [::1]:8765: connect: connection refused. Trying again in 16.777216s
INFO: error: dial tcp [::1]:8765: connect: connection refused. Trying again in 26.8435456s
INFO: error: dial tcp [::1]:8765: connect: connection refused. Trying again in 42.94967296sHelm values:
# Default values for Falco.
image:
registry: docker.io
repository: falcosecurity/falco-no-driver
tag: 0.34.0
pullPolicy: IfNotPresent
pullSecrets: []
docker:
enabled: false
socket: /var/run/docker.sock
containerd:
enabled: true
socket: /run/containerd/containerd.sock
crio:
enabled: false
socket: /run/crio/crio.sock
kubernetesSupport:
# Enables Kubernetes meta data collection via a connection to the Kubernetes API server.
enabled: true
# The apiAuth value is to provide the authentication method Falco should use to connect to the Kubernetes API.
# The argument's documentation from Falco is provided here for reference:
#
# <bt_file> | <cert_file>:<key_file[#password]>[:<ca_cert_file>], --k8s-api-cert <bt_file> | <cert_file>:<key_file[#password]>[:<ca_cert_file>]
# Use the provided files names to authenticate user and (optionally) verify the K8S API server identity.
# Each entry must specify full (absolute, or relative to the current directory) path to the respective file.
# Private key password is optional (needed only if key is password protected).
# CA certificate is optional. For all files, only PEM file format is supported.
# Specifying CA certificate only is obsoleted - when single entry is provided
# for this option, it will be interpreted as the name of a file containing bearer token.
# Note that the format of this command-line option prohibits use of files whose names contain
# ':' or '#' characters in the file name.
apiAuth: /var/run/secrets/kubernetes.io/serviceaccount/token
apiUrl: "https://$(KUBERNETES_SERVICE_HOST)"
# If true, only the current node (on which Falco is running) will be considered when requesting metadata of pods
# to the API server. Disabling this option may have a performance penalty on large clusters.
enableNodeFilter: true
resources:
# Although resources needed are subjective on the actual workload we provide
# a sane defaults ones. If you have more questions or concerns, please refer
# to #falco slack channel for more info about it
requests:
cpu: 100m
memory: 512Mi
limits:
cpu: 1000m
memory: 1024Mi
extraArgs: []
nodeSelector: {}
affinity: {}
rbac:
# Create and use rbac resources
create: true
podSecurityPolicy:
# Create a podSecurityPolicy
create: false
serviceAccount:
# Create and use serviceAccount resources
create: true
# Use this value as serviceAccountName
name:
annotations: {}
fakeEventGenerator:
enabled: false
args:
- run
- --loop
- ^syscall
replicas: 1
daemonset:
# Perform rolling updates by default in the DaemonSet agent
# ref: https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/
updateStrategy:
# You can also customize maxUnavailable or minReadySeconds if you
# need it
type: RollingUpdate
## Extra environment variables that will be pass onto deployment pods
env: {}
## Add aditional pod annotations on pods created by DaemonSet
podAnnotations: {}
# Additional labels to add to the pods:
# podLabels:
# key: value
podLabels: {}
# If is behind a proxy you can set the proxy server
proxy:
httpProxy:
httpsProxy:
noProxy:
# Set daemonset timezone
timezone: Europe/Amsterdam
# Set daemonset priorityClassName
priorityClassName:
ebpf:
# Enable eBPF support for Falco
enabled: true
path:
settings:
# Needed to enable eBPF JIT at runtime for performance reasons.
# Can be skipped if eBPF JIT is enabled from outside the container
hostNetwork: true
leastPrivileged:
# Constrain Falco with capabilities instead of running a privileged container.
# This option is only supported with the eBPF driver and a kernel >= 5.8.
# Ensure the eBPF driver is enabled (i.e., setting the `ebpf.enabled` option to true).
enabled: true
auditLog:
# If true, this will enable the Audit Log support in the chart.
# Note that the k8saudit plugin must be enabled and configured
# (see below).
enabled: false
# Please use the same port configured for the builtin k8saudit (usually 9765).
listenPort: 9765
# Enable a NodePort on the given port to listen on for the Audit Log.
# If false, no NodePort will be created.
nodePort: false
falco:
# File(s) or Directories containing Falco rules, loaded at startup.
# The name "rules_file" is only for backwards compatibility.
# If the entry is a file, it will be read directly. If the entry is a directory,
# every file in that directory will be read, in alphabetical order.
#
# falco_rules.yaml ships with the falco package and is overridden with
# every new software version. falco_rules.local.yaml is only created
# if it doesn't exist. If you want to customize the set of rules, add
# your customizations to falco_rules.local.yaml.
#
# The files will be read in the order presented here, so make sure if
# you have overrides they appear in later files.
rulesFile:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
# - /etc/falco/k8s_audit_rules.yaml
# - /etc/falco/aws_cloudtrail_rules.yaml
- /etc/falco/rules.d
# - /etc/falco/rules.optional.d
plugins:
- name: k8saudit
library_path: libk8saudit.so
init_config:
""
# maxEventBytes: 1048576
# sslCertificate: /etc/falco/falco.pem
open_params: "http://:9765/k8s-audit"
- name: cloudtrail
library_path: libcloudtrail.so
init_config: ""
open_params: ""
- name: json
library_path: libjson.so
init_config: ""
# Setting this list to empty ensures that the above plugins are *not*
# loaded and enabled by default. If you want to use the above plugins,
# set a meaningful init_config/open_params for the cloudtrail plugin
# and then change this to:
# load_plugins: [cloudtrail, json]
loadPlugins: []
# Watch config file and rules files for modification.
# When a file is modified, Falco will propagate new config,
# by reloading itself.
watchConfigFiles: true
# If true, the times displayed in log messages and output messages
# will be in ISO 8601. By default, times are displayed in the local
# time zone, as governed by /etc/localtime.
timeFormatISO8601: false
# Whether to output events in json or text
jsonOutput: false
# When using json output, whether or not to include the "output" property
# itself (e.g. "File below a known binary directory opened for writing
# (user=root ....") in the json output.
jsonIncludeOutputProperty: true
# When using json output, whether or not to include the "tags" property
# itself in the json output. If set to true, outputs caused by rules
# with no tags will have a "tags" field set to an empty array. If set to
# false, the "tags" field will not be included in the json output at all.
jsonIncludeTagsProperty: true
# Send information logs to stderr and/or syslog Note these are *not* security
# notification logs! These are just Falco lifecycle (and possibly error) logs.
logStderr: true
logSyslog: true
# Minimum log level to include in logs. Note: these levels are
# separate from the priority field of rules. This refers only to the
# log level of Falco's internal logging. Can be one of "emergency",
# "alert", "critical", "error", "warning", "notice", "info", "debug".
logLevel: info
# Minimum rule priority level to load and run. All rules having a
# priority more severe than this level will be loaded/run. Can be one
# of "emergency", "alert", "critical", "error", "warning", "notice",
# "informational", "debug".
priority: debug
# Whether or not output to any of the output channels below is
# buffered.
bufferedOutputs: false
# Falco uses a shared buffer between the kernel and userspace to pass
# system call information. When Falco detects that this buffer is
# full and system calls have been dropped, it can take one or more of
# the following actions:
# - ignore: do nothing (default when list of actions is empty)
# - log: log a DEBUG message noting that the buffer was full
# - alert: emit a Falco alert noting that the buffer was full
# - exit: exit Falco with a non-zero rc
#
# Notice it is not possible to ignore and log/alert messages at the same time.
#
# The rate at which log/alert messages are emitted is governed by a
# token bucket. The rate corresponds to one message every 30 seconds
# with a burst of one message (by default).
#
# The messages are emitted when the percentage of dropped system calls
# with respect the number of events in the last second
# is greater than the given threshold (a double in the range [0, 1]).
#
# For debugging/testing it is possible to simulate the drops using
# the `simulate_drops: true`. In this case the threshold does not apply.
syscallEventDrops:
threshold: .1
actions:
- log
- alert
rate: .03333
maxBurst: 1
# Falco uses a shared buffer between the kernel and userspace to receive
# the events (eg., system call information) in userspace.
#
# Anyways, the underlying libraries can also timeout for various reasons.
# For example, there could have been issues while reading an event.
# Or the particular event needs to be skipped.
# Normally, it's very unlikely that Falco does not receive events consecutively.
#
# Falco is able to detect such uncommon situation.
#
# Here you can configure the maximum number of consecutive timeouts without an event
# after which you want Falco to alert.
# By default this value is set to 1000 consecutive timeouts without an event at all.
# How this value maps to a time interval depends on the CPU frequency.
syscallEventTimeouts:
maxConsecutives: 1000
# Falco continuously monitors outputs performance. When an output channel does not allow
# to deliver an alert within a given deadline, an error is reported indicating
# which output is blocking notifications.
# The timeout error will be reported to the log according to the above log_* settings.
# Note that the notification will not be discarded from the output queue; thus,
# output channels may indefinitely remain blocked.
# An output timeout error indeed indicate a misconfiguration issue or I/O problems
# that cannot be recovered by Falco and should be fixed by the user.
#
# The "output_timeout" value specifies the duration in milliseconds to wait before
# considering the deadline exceed.
#
# With a 2000ms default, the notification consumer can block the Falco output
# for up to 2 seconds without reaching the timeout.
output_timeout: 2000
# A throttling mechanism implemented as a token bucket limits the
# rate of Falco notifications. This throttling is controlled by the following configuration
# options:
# - rate: the number of tokens (i.e. right to send a notification)
# gained per second. Defaults to 1.
# - max_burst: the maximum number of tokens outstanding. Defaults to 1000.
#
# With these defaults, Falco could send up to 1000 notifications after
# an initial quiet period, and then up to 1 notification per second
# afterward. It would gain the full burst back after 1000 seconds of
# no activity.
outputs:
rate: 1
maxBurst: 1000
# Where security notifications should go.
# Multiple outputs can be enabled.
syslogOutput:
enabled: true
# If keep_alive is set to true, the file will be opened once and
# continuously written to, with each output message on its own
# line. If keep_alive is set to false, the file will be re-opened
# for each output message.
#
# Also, the file will be closed and reopened if Falco is signaled with
# SIGUSR1.
fileOutput:
enabled: false
keepAlive: false
filename: ./events.txt
stdoutOutput:
enabled: true
# Falco contains an embedded webserver that can be used to accept K8s
# Audit Events. These config options control the behavior of that
# webserver. (By default, the webserver is enabled).
#
# The ssl_certificate is a combination SSL Certificate and corresponding
# key contained in a single file. You can generate a key/cert as follows:
#
# $ openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
# $ cat certificate.pem key.pem > falco.pem
# $ sudo cp falco.pem /etc/falco/falco.pem
#
# It also exposes a healthy endpoint that can be used to check if Falco is up and running
# By default the endpoint is /healthz
webserver:
enabled: true
listenPort: 8765
k8sHealthzEndpoint: /healthz
sslEnabled: false
sslCertificate: /etc/falco/certs/falco.pem
livenessProbe:
initialDelaySeconds: 60
timeoutSeconds: 5
periodSeconds: 15
readinessProbe:
initialDelaySeconds: 30
timeoutSeconds: 5
periodSeconds: 15
# Possible additional things you might want to do with program output:
# - send to a slack webhook:
# program: "\"jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX\""
# - logging (alternate method than syslog):
# program: logger -t falco-test
# - send over a network connection:
# program: nc host.example.com 80
# If keep_alive is set to true, the program will be started once and
# continuously written to, with each output message on its own
# line. If keep_alive is set to false, the program will be re-spawned
# for each output message.
#
# Also, the program will be closed and reopened if Falco is signaled with
# SIGUSR1.
programOutput:
enabled: false
keepAlive: false
program: mail -s "Falco Notification" someone@example.com
# program: |
# jq 'if .priority == "Emergency" or .priority == "Critical" or .priority == "Error" then
# { attachments: [{ text: .output, color: "danger" }]}
# elif .priority == "Warning" or .priority == "Notice" then
# { attachments: [{ text: .output, color: "warning" }]}
# elif .priority == "Informational" then
# { attachments: [{ text: .output, color: "good" }]}
# else
# { attachments: [{ text: .output }]}
# end' | curl -d @- -X POST https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxx
httpOutput:
enabled: false
# When set, this will override an auto-generated URL which matches the falcosidekick Service.
# When including Falco inside a parent helm chart, you must set this since the auto-generated URL won't match (#280).
url: ""
userAgent: "falcosecurity/falco"
# Falco supports running a gRPC server with two main binding types
# 1. Over the network with mandatory mutual TLS authentication (mTLS)
# 2. Over a local unix socket with no authentication
# By default, the gRPC server is disabled, with no enabled services (see grpc_output)
# please comment/uncomment and change accordingly the options below to configure it.
# Important note: if Falco has any troubles creating the gRPC server
# this information will be logged, however the main Falco daemon will not be stopped.
# gRPC server over network with (mandatory) mutual TLS configuration.
# This gRPC server is secure by default so you need to generate certificates and update their paths here.
# By default the gRPC server is off.
# You can configure the address to bind and expose it.
# By modifying the threadiness configuration you can fine-tune the number of threads (and context) it will use.
grpc:
enabled: false
threadiness: 0
# gRPC unix socket with no authentication
unixSocketPath: "unix:///var/run/falco/falco.sock"
# gRPC over the network (mTLS) / required when unixSocketPath is empty
listenPort: 5060
privateKey: "/etc/falco/certs/server.key"
certChain: "/etc/falco/certs/server.crt"
rootCerts: "/etc/falco/certs/ca.crt"
# gRPC output service.
# By default it is off.
# By enabling this all the output events will be kept in memory until you read them with a gRPC client.
# Make sure to have a consumer for them or leave this disabled.
grpcOutput:
enabled: false
# Container orchestrator metadata fetching params
metadataDownload:
maxMb: 100
chunkWaitUs: 1000
watchFreqSec: 1
customRules:
{}
# Although Falco comes with a nice default rule set for detecting weird
# behavior in containers, our users are going to customize the run-time
# security rule sets or policies for the specific container images and
# applications they run. This feature can be handled in this section.
#
# Example:
#
# rules-traefik.yaml: |-
# [ rule body ]
# certificates used by webserver and grpc server
# paste certificate content or use helm with --set-file
# or use existing secret containing key, crt, ca as well as pem bundle
certs:
existingSecret: ""
server:
key: ""
crt: ""
ca:
crt: ""
# Allow Falco to run on Kubernetes 1.6 masters.
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
scc:
# true here enabled creation of Security Context Constraints in Openshift
create: true
# Add initContainers to Falco pod
extraInitContainers:
- name: driver-loader
image: docker.io/falcosecurity/falco-driver-loader:latest
imagePullPolicy: Always
volumeMounts:
- mountPath: /host/proc
name: proc-fs
readOnly: true
- mountPath: /host/boot
name: boot-fs
readOnly: true
- mountPath: /host/lib/modules
name: lib-modules
- mountPath: /host/usr
name: usr-fs
readOnly: true
- mountPath: /host/etc
name: etc-fs
readOnly: true
- mountPath: /root/.falco
name: driver-fs
env:
- name: FALCO_BPF_PROBE
value: null
# Add extra volumes to Falco daemonset
extraVolumes:
- name: driver-fs
hostPath: /root/.falco
# - name: optional-rules-volume
# configMap:
# name: falco-rules-optional
# optional: true
# items:
# - key: falco_rules.optional.yaml
# path: falco_rules.optional.yaml
# Add extra volumeMounts to Falco container in Falco daemonset
extraVolumeMounts:
- mountPath: /root/.falco
name: driver-fs
# - mountPath: /etc/falco/rules.optional.d
# name: optional-rules-volume
falcosidekick:
# enable falcosidekick deployment
enabled: false
fullfqdn: false
# for configuration values, see https://github.com/falcosecurity/charts/blob/master/falcosidekick/values.yamlkernelrelease: 5.15.0-60-generic
kernelversion: 66
target: ubuntu-generic
output:
module: /tmp/falco.ko
probe: /tmp/falco-bpf.o
driverversion: master$ ./driverkit kubernetes -c ubuntu.yaml
INFO using config file file=ubuntu.yaml
INFO driver building, it will take a few seconds processor=kubernetes
W0217 10:47:07.692085 777722 warnings.go:70] spec.imagePullSecrets[0].name: invalid empty name ""
W0217 10:47:07.692176 777722 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "driverkit-18479752-1c39-4558-8b8f-2ab4d348a080" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "driverkit-18479752-1c39-4558-8b8f-2ab4d348a080" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "driverkit-18479752-1c39-4558-8b8f-2ab4d348a080" must set securityContext.runAsNonRoot=true), runAsUser=0 (pod must not set runAsUser=0), seccompProfile (pod or container "driverkit-18479752-1c39-4558-8b8f-2ab4d348a080" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
INFO start downloading module and probe from pod org.falcosecurity/driverkit-uid=18479752-1c39-4558-8b8f-2ab4d348a080
INFO Kernel Module extraction successful
INFO Probe Module extraction successful
INFO completed downloading from pod org.falcosecurity/driverkit-uid=18479752-1c39-4558-8b8f-2ab4d348a080
$ sudo cp falco-bpf.o /root/.falco/falco-bpf.o
$ sudo cp falco.ko /root/.falco/falco.koHi! It seems like driverkit is working, and Falco can also find a prebuilt driver:
- Trying to download a prebuilt falco module from https://download.falco.org/driver/4.0.0%2Bdriver/x86_64/falco_ubuntu-generic_5.15.0-60-generic_66.ko
- Download succeeded
- Success: falco module found and inserted
But it seems like new falcoctl tool is failing to talk with Falco. See https://falco.org/blog/falco-0-34-0/ and https://falco.org/blog/rules-helm-chart-3-0-0/ for more infos. @alacuku any idea? Btw i think we can close the issue on driverkit, and perhaps opening a new one on Falco, if you agree. Thank you!
Jeroen0494
commented
1 year ago Hi,
Also, I specified it should use eBPF, but it loads a .ko module instead of the eBPF program. Is there something going wrong here?
FedeDP
commented
1 year ago Falco driver loader is trying to load the kmod indeed:
- Running falco-driver-loader for: falco version=0.34.0, driver version=4.0.0+driver, arch=x86_64, kernel release=5.15.0-60-generic, kernel version=66
- Running falco-driver-loader with: driver=module, compile=yes, download=yes
Again, i suggest to open an issue on Falco that will get more visibility than this one! And this one is fixed i guess! ;)
alacuku
commented
1 year ago @alacuku any idea? Btw i think we can close the issue on driverkit, and perhaps opening a new one on Falco, if you agree. Thank you!
Based on this logs:
jeroen@mediaserver:~/kubernetes/k3s/falco$ k logs -n falco falco-rmmx5 -c falco
Fri Feb 17 09:42:56 2023: Falco version: 0.34.0 (x86_64)
Fri Feb 17 09:42:56 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Feb 17 09:42:56 2023: Loading rules from file /etc/falco/falco_rules.yaml
Error: Could not create inotify handlerIt seems that falco did not start at all. Falcoctl tries to connect to falco, and if it is not up and running it fails after a preset number of retries.
FedeDP
commented
1 year ago Uh you are right, thanks Aldo, didn't even notice about that! @Jeroen0494 yep this needs a proper issue on Falco, seems like a bad bug(?)!
alacuku
commented
1 year ago @Jeroen0494, what is the version of the helm chart you are using to deploy Falco? It seems that the values.yaml file you posted is outdated.
FedeDP
commented
1 year ago Ps: inotify_init can just fail for a bunch of reasons (https://man7.org/linux/man-pages/man2/inotify_init.2.html):
EMFILE The user limit on the total number of inotify instances
has been reached.
EMFILE The per-process limit on the number of open file
descriptors has been reached.
ENFILE The system-wide limit on the total number of open files
has been reached.
ENOMEM Insufficient kernel memory is available.I'll try again with an up-to-date helm chart. In the mean time, here are my limits:
$ ulimit -a
real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 61679
max locked memory (kbytes, -l) 1989824
max memory size (kbytes, -m) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 61679
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimitedI receive the same error when I don't specify a values file, but I found the fix:
jeroen@mediaserver:~/kubernetes/k3s/falco$ k logs -n falco falco-bbmn8
Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falco-driver-loader (init), falcoctl-artifact-install (init)
Fri Feb 17 10:21:29 2023: Falco version: 0.34.0 (x86_64)
Fri Feb 17 10:21:29 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Feb 17 10:21:29 2023: Loading rules from file /etc/falco/falco_rules.yaml
Error: Could not create inotify handler
jeroen@mediaserver:~/kubernetes/k3s/falco$ cat /proc/sys/fs/inotify/max_user_instances
128
jeroen@mediaserver:~/kubernetes/k3s/falco$ sudo -i
[sudo] password for jeroen:
root@mediaserver:~# echo 256 > /proc/sys/fs/inotify/max_user_instances
root@mediaserver:~#
logout
jeroen@mediaserver:~/kubernetes/k3s/falco$ cat /proc/sys/fs/inotify/max_user_instances
256
jeroen@mediaserver:~/kubernetes/k3s/falco$ k delete pod -n falco falco-bbmn8
pod "falco-bbmn8" deleted
jeroen@mediaserver:~/kubernetes/k3s/falco$ k get pod -n falco -w
NAME READY STATUS RESTARTS AGE
falco-jf22t 0/2 Init:1/2 0 5s
falco-jf22t 0/2 PodInitializing 0 7s
falco-jf22t 1/2 Running 0 10s
falco-jf22t 2/2 Running 0 45s
^Cjeroen@mediaserver:~/kubernetes/k3s/falco$ k logs -n falco falco-jf22t
Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falco-driver-loader (init), falcoctl-artifact-install (init)
Fri Feb 17 10:23:58 2023: Falco version: 0.34.0 (x86_64)
Fri Feb 17 10:23:58 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Feb 17 10:23:58 2023: Loading rules from file /etc/falco/falco_rules.yaml
Fri Feb 17 10:23:58 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Fri Feb 17 10:23:58 2023: Starting health webserver with threadiness 4, listening on port 8765
Fri Feb 17 10:23:58 2023: Enabled event sources: syscall
Fri Feb 17 10:23:58 2023: Opening capture with Kernel module
10:24:00.697220059: Error File below /etc opened for writing (user= user_loginuid=-1 command=history_event-z /etc/zfs/zed.d/history_event-zfs-list-cacher.sh pid=973555 parent=<NA> pcmdline=<NA> file=/etc/zfs/zfs-list.cache/rpool program=history_event-z gparent=<NA> ggparent=<NA> gggparent=<NA> container_id=host image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=host
10:24:01.602694769: Notice Network tool launched in container (user=root user_loginuid=-1 command=nc crowdsec-service.crowdsec 8080 pid=973812 parent_process=sh container_id=edf8eb5c0d09 container_name=wait-for-lapi image=docker.io/library/busybox:1.28) k8s.ns=crowdsec k8s.pod=crowdsec-agent-f67m4 container=edf8eb5c0d09
10:24:07.614822189: Notice Network tool launched in container (user=root user_loginuid=-1 command=nc crowdsec-service.crowdsec 8080 pid=974295 parent_process=sh container_id=edf8eb5c0d09 container_name=wait-for-lapi image=docker.io/library/busybox:1.28) k8s.ns=crowdsec k8s.pod=crowdsec-agent-f67m4 container=edf8eb5c0d09
10:24:07.762422107: Error File below /etc opened for writing (user=root user_loginuid=-1 command=history_event-z /etc/zfs/zed.d/history_event-zfs-list-cacher.sh pid=974358 parent=zed pcmdline=zed -F file=/etc/zfs/zfs-list.cache/rpool program=history_event-z gparent=systemd ggparent=<NA> gggparent=<NA> container_id=host image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=host
10:24:08.297478520: Error File below /etc opened for writing (user=root user_loginuid=-1 command=history_event-z /etc/zfs/zed.d/history_event-zfs-list-cacher.sh pid=974439 parent=zed pcmdline=zed -F file=/etc/zfs/zfs-list.cache/rpool program=history_event-z gparent=systemd ggparent=<NA> gggparent=<NA> container_id=host image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=host
10:24:08.727443912: Error File below /etc opened for writing (user=root user_loginuid=-1 command=history_event-z /etc/zfs/zed.d/history_event-zfs-list-cacher.sh pid=974358 parent=zed pcmdline=zed -F file=/etc/zfs/zfs-list.cache/rpool program=history_event-z gparent=systemd ggparent=<NA> gggparent=<NA> container_id=host image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=host
10:24:09.223476753: Notice Unexpected connection to K8s API Server from container (command=keel pid=974411 k8s.ns=keel k8s.pod=keel-777db7f87f-rz4n4 container=2fe7570bd096 image=docker.io/keelhq/keel:0.17.0-rc1 connection=10.233.105.131:57710->10.43.0.1:443)
10:24:09.223739698: Notice Unexpected connection to K8s API Server from container (command=keel pid=974411 k8s.ns=keel k8s.pod=keel-777db7f87f-rz4n4 container=2fe7570bd096 image=docker.io/keelhq/keel:0.17.0-rc1 connection=10.233.105.131:57730->10.43.0.1:443)
10:24:09.223781943: Notice Unexpected connection to K8s API Server from container (command=keel pid=974411 k8s.ns=keel k8s.pod=keel-777db7f87f-rz4n4 container=2fe7570bd096 image=docker.io/keelhq/keel:0.17.0-rc1 connection=10.233.105.131:57724->10.43.0.1:443)
10:24:09.225446769: Notice Unexpected connection to K8s API Server from container (command=keel pid=974411 k8s.ns=keel k8s.pod=keel-777db7f87f-rz4n4 container=2fe7570bd096 image=docker.io/keelhq/keel:0.17.0-rc1 connection=10.233.105.131:57738->10.43.0.1:443)
10:24:09.666783451: Error File below /etc opened for writing (user=root user_loginuid=-1 command=history_event-z /etc/zfs/zed.d/history_event-zfs-list-cacher.sh pid=974439 parent=zed pcmdline=zed -F file=/etc/zfs/zfs-list.cache/rpool program=history_event-z gparent=systemd ggparent=<NA> gggparent=<NA> container_id=host image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=host
10:24:13.634813941: Notice Network tool launched in container (user=root user_loginuid=-1 command=nc crowdsec-service.crowdsec 8080 pid=974630 parent_process=sh container_id=edf8eb5c0d09 container_name=wait-for-lapi image=docker.io/library/busybox:1.28) k8s.ns=crowdsec k8s.pod=crowdsec-agent-f67m4 container=edf8eb5c0d09Need to add the following sysctl:
fs.inotify.max_user_instances = 256Great find man! :rocket:
Jeroen0494
commented
1 year ago Now it's just a matter of making myself familiar with Falco, I consider this issue to be resolved.
Thanks you for all your help!
Describe the bug Driverkit fails to build on Ubuntu 22.04 because the kernel is build with GCC 11, and the builder image has GCC 10.
How to reproduce it Driverkit download:
ubuntu.yaml:
Build with
./driverkit kubernetes -c ubuntu.yamlOutput:
Environment