search

falcosecurity / falco

Cloud Native Runtime Security
https://falco.org
Apache License 2.0
7.4k stars 902 forks source link

Does Falco support running via K3S on CM4? Pods error out. #3027

Closed MikeCockrem closed 4 months ago

MikeCockrem commented 9 months ago

Falco installed from Helm fails to come to the ready state on K3S running on Raspberry Pi 4 (CM4) It seems the ebfp module can't be built, I'm hoping someone can advise me on a way forward - from the logs it seems perhaps it may not be able to find the raspbian specific kernel headers?

Following the guide here I ran: helm upgrade falco -n falco --set tty=true falcosecurity/falco --set falcosidekick.enabled=true --set falcosidekick.config.slack.minimumpriority=notice --set falcosidekick.config.slack.webhookurl=$(base64 --decode <<< "webhook-goes-here") --set falcosidekick.config.customfields="user:redacted" --set driver.kind=ebpf

After some time this is the output:

 $ kubectl get pods -n falco
 NAME                                   READY   STATUS                  RESTARTS          AGE
 falco-falcosidekick-6894b7c676-kh8gb   1/1     Running                 0                 16h
 falco-falcosidekick-6894b7c676-2fnns   1/1     Running                 0                 16h
 falco-ppnnq                            0/2     Init:CrashLoopBackOff   200 (4m30s ago)   16h
 falco-r6nqm                            0/2     Init:CrashLoopBackOff  200 (3m47s ago)   16h
 falco-rpxlr                                0/2     Init:Error          199 (5m20s ago)   16h
 falco-z54np                            0/2     Init:Error              201 (5m18s ago)   16h_

log files are thus:

$ kubectl -n falco logs daemonsets/falco -c falco-driver-loader
Found 4 pods, using pod/falco-ppnnq
* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.36.2, driver version=6.0.1+driver, arch=aarch64, kernel release=6.1.32-v8+, kernel version=1656
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
mount: /sys/kernel/debug: permission denied.
* Filename 'falco_raspbian_6.1.32-v8+_1656.o' is composed of:
 - driver name: falco
 - target identifier: raspbian
 - kernel release: 6.1.32-v8+
 - kernel version: 1656
* Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/6.0.1%2Bdriver/aarch64/falco_raspbian_6.1.32-v8%2B_1656.o
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco eBPF probe
* Trying to compile the eBPF probe (falco_raspbian_6.1.32-v8+_1656.o)
expr: syntax error: unexpected argument '1'
make[1]: *** /lib/modules/6.1.32-v8+/build: No such file or directory.  Stop.
make: *** [Makefile:38: all] Error 2
mv: cannot stat '/usr/src/falco-6.0.1+driver/bpf/probe.o': No such file or directory
Unable to load the falco eBPF probe

Expected behaviour

Pods to come to the running state

Screenshots

Environment

Additional context I've tried adding the kernel headers and build environment. The folder the error complains about does exist:

ls /lib/modules/6.1.32-v8+/
modules.alias      modules.builtin            modules.builtin.bin      modules.dep      modules.devname  modules.softdep  modules.symbols.bin
kernel  modules.alias.bin  modules.builtin.alias.bin  modules.builtin.modinfo  modules.dep.bin  modules.order    modules.symbols

Very cool project I'd like to start learning so I'd appreciate any pointers. Cheers.

Andreagit97 commented 9 months ago

ei @MikeCockrem have you tried the modern-bpf driver? It seems you have a compatible kernel driver version=6.0.1+driver

The command is the same you posted but you need to replace driver.kind=ebpf with driver.kind=modern-bpf

helm upgrade falco -n falco --set tty=true falcosecurity/falco --set falcosidekick.enabled=true --set falcosidekick.config.slack.minimumpriority=notice --set falcosidekick.config.slack.webhookurl=$(base64 --decode <<< "webhook-goes-here") --set falcosidekick.config.customfields="user:redacted" --set driver.kind=modern-bpf
MikeCockrem commented 9 months ago

Thanks, I've tried that out, however I get the following error:

$ kgp -n falco
NAME                                   READY   STATUS             RESTARTS        AGE
falco-falcosidekick-6894b7c676-kh8gb   1/1     Running            0               20h
falco-falcosidekick-6894b7c676-2fnns   1/1     Running            0               20h
falco-thpmh                            1/2     CrashLoopBackOff   17 (115s ago)   63m
falco-rmwbt                            0/2     CrashLoopBackOff   29 (110s ago)   63m
falco-rqtcv                            0/2     CrashLoopBackOff   29 (95s ago)    64m
falco-fp4w4                            0/2     CrashLoopBackOff   29 (90s ago)    63m

kubectl -n falco logs daemonsets/falco -c falco-driver-loader
Found 4 pods, using pod/falco-rqtcv
error: container falco-driver-loader is not valid for pod falco-rqtcv

kubectl logs -n falco falco-thpmh
Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falcoctl-artifact-install (init)
Mon Jan 22 11:55:56 2024: Falco version: 0.36.2 (aarch64)
Mon Jan 22 11:55:56 2024: Falco initialized with configuration file: /etc/falco/falco.yaml
Mon Jan 22 11:55:56 2024: Loading rules from file /etc/falco/falco_rules.yaml
Mon Jan 22 11:55:56 2024: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Mon Jan 22 11:55:56 2024: Starting health webserver with threadiness 4, listening on port 8765
Mon Jan 22 11:55:56 2024: Loaded event sources: syscall
Mon Jan 22 11:55:56 2024: Enabled event sources: syscall
Mon Jan 22 11:55:56 2024: Opening 'syscall' source with modern BPF probe.
Mon Jan 22 11:55:56 2024: One ring buffer every '2' CPUs.
libpman: prog 'BPF_TRACE_RAW_TP' is not supported <-------------------------------------------<
Mon Jan 22 11:55:56 2024: An error occurred in an event source, forcing termination...
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
Error: Initialization issues during scap_init

I would guess that means support for modern-bpf is not compiled into the kernel for pi?

Andreagit97 commented 9 months ago

uhm got it, yes the modern bpf is not supported since your kernel misses some key features needed to run it. Not sure what is causing your issue, @FedeDP any idea?

FedeDP commented 9 months ago

I've tried adding the kernel headers and build environment. The folder the error complains about does exist:

Note that it complains about the build folder: /lib/modules/6.1.32-v8+/build and i don't see it being listed in your ls output.

MikeCockrem commented 9 months ago

I've tried adding the kernel headers and build environment. The folder the error complains about does exist:

Note that it complains about the build folder: /lib/modules/6.1.32-v8+/build and i don't see it being listed in your ls output.

Thanks for the pointer, I messed around trying to install the kernel sources and have now come up against this error:

Trying to compile the eBPF probe (falco_raspbian_6.1.32-v8+_1656.o) 
expr: syntax error: unexpected argument '1'
make[1]: *** No targets specified and no makefile found.  Stop.
make: *** [Makefile:38: all] Error 2
mv: cannot stat '/usr/src/falco-6.0.1+driver/bpf/probe.o': No such file or directory
Unable to load the falco eBPF probe
Stream closed EOF for falco/falco-gcc4m (falco-driver-loader)

I'll try and have another look at it tomorrow.

poiana commented 6 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

poiana commented 5 months ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

poiana commented 4 months ago

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community. /close

poiana commented 4 months ago

@poiana: Closing this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/3027#issuecomment-2198243355): >Rotten issues close after 30d of inactivity. > >Reopen the issue with `/reopen`. > >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Provide feedback via https://github.com/falcosecurity/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.