search

falcosecurity / libs

libsinsp, libscap, the kernel module driver, and the eBPF driver sources
https://falcosecurity.github.io/libs/
Apache License 2.0
222 stars 162 forks source link

eBPF Probe Compilation Fails on Amazon EKS BottleRocket #706

Closed HarishKM7 closed 1 year ago

HarishKM7 commented 1 year ago

Bug Description

The falco-driver-loader init container of the falco pod tries to compile an eBPF probe & fails.

Steps to Reproduce

helm install falco falco/falco --set driver.kind=ebpf

Check logs of the falco-driver-loader init container of the falco pod.

Expected Behaviour

Actual Behaviour

The falco-driver-loader init container of the falco pod fails with these logs:

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.33.0, driver version=3.0.1+driver, arch=x86_64, kernel release=5.10.135, kernel version=1
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
mount: /sys/kernel/debug: permission denied.
* Filename 'falco_bottlerocket_5.10.135_1.o' is composed of:
 - driver name: falco
 - target identifier: bottlerocket
 - kernel release: 5.10.135
 - kernel version: 1
* Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/3.0.1%2Bdriver/x86_64/falco_bottlerocket_5.10.135_1.o
curl: (22) The requested URL returned error: 404 
Unable to find a prebuilt falco eBPF probe
* Trying to compile the eBPF probe (falco_bottlerocket_5.10.135_1.o)
exec-cmd.c:210:1: fatal error: opening dependency file /host/usr/src/kernels/5.10.135/tools/objtool/.exec-cmd.o.d: Permission denied
 }
 ^
compilation terminated.
make[5]: *** [/host/usr/src/kernels/5.10.135/tools/build/Makefile.build:97: /host/usr/src/kernels/5.10.135/tools/objtool/exec-cmd.o] Error 1
make[4]: *** [Makefile:59: /host/usr/src/kernels/5.10.135/tools/objtool/libsubcmd-in.o] Error 2
make[3]: *** [Makefile:68: /host/usr/src/kernels/5.10.135/tools/objtool/libsubcmd.a] Error 2
make[2]: *** [Makefile:68: objtool] Error 2
make[1]: *** [Makefile:1934: tools/objtool] Error 2
make: *** [Makefile:38: all] Error 2
mv: cannot stat '/usr/src/falco-3.0.1+driver/bpf/probe.o': No such file or directory
Unable to load the falco eBPF probe

Relevant info from a BottleRocket EKS worker node:

# SSM to EKS worker node
[ssm-user@control]$ enter-admin-container
[root@admin]# sudo sheltie
bash-5.1# ls -al /usr/src/kernels/5.10.135/tools/objtool/ | grep exec
-rw-r--r--. 1 root root    5284 Nov  9 05:14 .exec-cmd.o.cmd
-rw-r--r--. 1 root root  156576 Nov  9 05:14 exec-cmd.o

Environment

Falco Helm Chart Version: 2.2.0 Falco App Version: 0.33.0

Amazon EKS Kubernetes Version: 1.23 BottleRocket AMI Version: 1.10.1-5d27ae74

# SSM to EKS worker node
[ssm-user@control]$ enter-admin-container
[root@admin]# sudo sheltie
bash-5.1# ls /etc/*-release
/etc/os-release
bash-5.1# cat /etc/os-release
NAME=Bottlerocket
ID=bottlerocket
VERSION="1.10.1 (aws-k8s-1.23)"
PRETTY_NAME="Bottlerocket OS 1.10.1 (aws-k8s-1.23)"
VARIANT_ID=aws-k8s-1.23
VERSION_ID=1.10.1
BUILD_ID=5d27ae74
HOME_URL="https://github.com/bottlerocket-os/bottlerocket"
SUPPORT_URL="https://github.com/bottlerocket-os/bottlerocket/discussions"
BUG_REPORT_URL="https://github.com/bottlerocket-os/bottlerocket/issues"
bash-5.1# uname -a
Linux ip-10-0-0-183.eu-west-1.compute.internal 5.10.135 #1 SMP Tue Oct 18 00:05:25 UTC 2022 x86_64 GNU/Linux
maynardflies commented 1 year ago

Having the same issue. I was able to go into the admin container in BottleRocket, pull the source, and build thru cmake, and then pull the precompiled probe.o from an s3 bucket (which you can check out here if so desired ), but then I got the following error

Error: invalid filler name: sys_epoll_create_e

And it crashloops. I see a similar issue from a while ago on an AL2 build that was fixed by this PR, so hopeful that this can be resolved as well

FedeDP commented 1 year ago

Hi! Sorry for the long delay! @maynardflies i think you built libs master and you want to run it against Falco 0.33 libs? We added sys_epoll_create support in the meantime, but Falco isn't built with the libs that support it. So, you should build driver at the 3.0.1+driver tag, that is the one used by Falco 0.33.

@HarishKM7 weird! Given your output, it seems like dep file .exec-cmd.o.d is not present:

exec-cmd.c:210:1: fatal error: opening dependency file /host/usr/src/kernels/5.10.135/tools/objtool/.exec-cmd.o.d: Permission denied }

bash-5.1# ls -al /usr/src/kernels/5.10.135/tools/objtool/ | grep exec -rw-r--r--. 1 root root 5284 Nov 9 05:14 .exec-cmd.o.cmd -rw-r--r--. 1 root root 156576 Nov 9 05:14 exec-cmd.o

But i never saw this issue before. I will play around it to see if we need any fix!

FedeDP commented 1 year ago

Note: we do not support pre-built drivers for Bottlerocket, since it is unsupported on kernel-crawler and driverkit yet.

PS: we will gladly accept any PR in that direction though :)

FedeDP commented 1 year ago

/assign

FedeDP commented 1 year ago

Did you read https://github.com/bottlerocket-os/bottlerocket/issues/862? It seems like it answers many questions!

FedeDP commented 1 year ago

And this comment too: https://github.com/bottlerocket-os/bottlerocket/issues/2275#issuecomment-1186315503

FedeDP commented 1 year ago

I started the work to add support for bottlerocket to our infra: https://github.com/falcosecurity/kernel-crawler/pull/79. Basically, we need 3 pieces glued together:

I will keep you updated about any news!

maynardflies commented 1 year ago

Sorry about the delay on this answer. Yes I have read this thread many times :) the results don't match it anymore, presumably because many versions of both BR and Falco have been released since Mar2020-Jan2021 and that functionality has broken

On Tue, Dec 6, 2022 at 8:37 AM Federico Di Pierro @.***> wrote:

Did you read bottlerocket-os/bottlerocket#862 https://github.com/bottlerocket-os/bottlerocket/issues/862? It seems like it answers many questions!

— Reply to this email directly, view it on GitHub https://github.com/falcosecurity/libs/issues/706#issuecomment-1339401557, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD2HUB7IENODOX3J5PKPUELWL46STANCNFSM6AAAAAAR3C3FZM . You are receiving this because you were mentioned.Message ID: @.***>

maynardflies commented 1 year ago

Not sure I saw this one

On Tue, Dec 6, 2022 at 8:43 AM Federico Di Pierro @.***> wrote:

And this comment too: bottlerocket-os/bottlerocket#2275 (comment) https://github.com/bottlerocket-os/bottlerocket/issues/2275#issuecomment-1186315503

— Reply to this email directly, view it on GitHub https://github.com/falcosecurity/libs/issues/706#issuecomment-1339408359, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD2HUB6WFEBVQX2MMMS4AK3WL47JRANCNFSM6AAAAAAR3C3FZM . You are receiving this because you were mentioned.Message ID: @.***>

FedeDP commented 1 year ago

The aforementioned comment (https://github.com/bottlerocket-os/bottlerocket/issues/2275#issuecomment-1186315503) is coming from a July thread, so not that far away(possibly with Falco 0.33)!

maynardflies commented 1 year ago

Yes and I can confirm with that setup, the build does work (albeit with some compilation warnings in falco-driver-loader) and the pods come up clean in the helm chart approach. Obviously we want to try and bake this straight into the machine image to avoid privileged pods as much as possible but this is a good step forward

FedeDP commented 1 year ago

Given latest kernel-crawler output: https://github.com/falcosecurity/kernel-crawler/pull/79#issuecomment-1348155276, i was able to correctly build kmod and eBPF probe using patched driverkit: https://github.com/falcosecurity/driverkit/pull/239.

I also updated Falco driver-loader script to correctly manage bottlerocket: https://github.com/falcosecurity/falco/pull/2318.

Everything is still wip because i've yet to test everything on a real bottlerocket instance.

FedeDP commented 1 year ago

Next kernel-crawler run will discover bottlerocket kernels; next, once driverkit PR is merged, we will need to tag a new driverkit release, bump driverkit in test-infra and add test-infra config for bottlerocket jobs ;)

FedeDP commented 1 year ago

We now have bottlerocket entries in kernel-crawler generated json: https://falcosecurity.github.io/kernel-crawler/?arch=x86_64&target=BottleRocket.

FedeDP commented 1 year ago

OK everything is now in place! :rocket: Drivers will start being built next monday, but your falco-driver-loader script won't be able to retrieve them because some changes were needed to it. You will need Falco 0.34 that is expected to be released around end of january ;)

maynardflies commented 1 year ago

awesome news! We will absolutely check it out as soon as 0.34 drops :) Thanks for the great work!

FedeDP commented 1 year ago

Falco 0.34 is out, and should work out of the box on Bottlerocket! :rocket: If you are able to test, we might want to close this one ;)

FedeDP commented 1 year ago

(You can see currently provided driver here: https://download.falco.org/driver/site/index.html?lib=4.0.0%2Bdriver&target=bottlerocket&arch=all&kind=all)

FedeDP commented 1 year ago

@HarishKM7 any news? :) Thank you!

HarishKM7 commented 1 year ago

Thanks for the fix everyone. Unfortunately, I won't be able to test it at least for the next few weeks. I'll try to test & reply as soon as I can. 🙂

HarishKM7 commented 1 year ago

Falco Helm chart version 3.1.2 installed successfully on Amazon EKS 1.24 with Bottlerocket 1.12 worker nodes & these Helm value overrides:

driver:
  kind: ebpf
falcosidekick:
  enabled: true
  webui:
    enabled: true
    ingress:
      enabled: true
      hosts:
      - host: ...
        paths:
        - path: /