search

falcosecurity / falco

Cloud Native Runtime Security
https://falco.org
Apache License 2.0
7.23k stars 893 forks source link

Debian pre-built modules failing #2625

Closed glpfloss closed 1 year ago

glpfloss commented 1 year ago

Describe the bug

Hello,

After the release of v0.35.0 addressing this other bug affecting Debian #2374, we realized the pre-built modules are failing when loading them under Debian, raising this error during Kubernetes init process (falco-driver-loader)

* Found a prebuilt falco module at /root/.falco/5.0.1+driver/x86_64/falco_debian_4.19.282-1-amd64_1.ko, loading it insmod: ERROR: could not insert module /root/.falco/5.0.1+driver/x86_64/falco_debian_4.19.282-1-amd64_1.ko: Invalid module format

This message can be observed in the OS dmesg falco: disagrees about version of symbol module_layout

How to reproduce it

Deploy https://github.com/falcosecurity/charts/tree/master/falco helm chart in a Kubernetes cluster using nodes with Debian (10 or 11)

Expected behaviour

The module should load ok

Environment

Additional context

Issue #2374 is related, as https://github.com/falcosecurity/falco/issues/2374#issuecomment-1409850595 is having the same problem after the fix

Thanks in advance, best regards

jasondellaluce commented 1 year ago

cc @falcosecurity/driverkit-maintainers

FedeDP commented 1 year ago

Hi! Thanks for opening this issue! We were indeed investigating something similar on ubuntu-aws on arm64 though.
I don't think it has anything to do with driverkit; most probably we are messing up the driver to be downloaded given the weird naming scheme used by debian (and ubuntu-aws too!). Of course, given #2374 fixed the issue for multiple users, it seems weird that we are still parsing the uname output wrong.

Linux kube1 4.19.0-24-amd64 #1 SMP Debian 4.19.282-1 (2023-04-29) x86_64 GNU/Linux

Care to try using ebpf instead?

FedeDP commented 1 year ago

For an ubuntu-aws example, see this: We got both:

driverkit configs (note: the latter is not even built because our test-infra regex is not smart enough), yet we resolve ubuntu-aws 22.04.1 to the former (ie: ubuntu-aws_5.19.0-1025-aws_26.ko) and it is giving the exact same issue. This is something @therealbobo is working on.
I think it might deserve a bugfix release, since it will involve touching falco-driver-loader for sure.

Your problem instead seems a bit weirder because, as i said, we had people testing the fix and it worked fine; unless there are multiple debian 4.19.282-1 in this case too, i don't get why it should fail. But i don't get why these distros hate us so much :laughing:

FedeDP commented 1 year ago

Another possibility is that we are using the wrong headers for some debian builds: https://falcosecurity.github.io/kernel-crawler/?arch=x86_64&target=Debian. For example, if i search 4.19. there, i see that headers for:

Driverkit downloads all of them: https://github.com/falcosecurity/driverkit/blob/master/pkg/driverbuilder/builder/templates/debian.sh#L18, then uses a regex to match the correct one: https://github.com/falcosecurity/driverkit/blob/master/pkg/driverbuilder/builder/templates/debian.sh#L30, that resolves to KernelHeadersPattern = "linux-headers-*" + kr.Architecture.String().
So, i think it might be trying to use eg: an rt configuration instead of the normal one. That would cause a breakage.

I will try a couple of things and come back later with more info!

FedeDP commented 1 year ago

So, i made some tests:

Using driverkit v0.13.0 (the one used by test-infra to build drivers), i re-built the kmod for your debian kernel; md5sums match:

7ecb5718117b373f6c58148bd1dc2fdf  /tmp/falco_debian_4.19.282-1-amd64_1.ko
7ecb5718117b373f6c58148bd1dc2fdf  falco_debian_4.19.282-1-amd64_1.ko

(the /tmp/ one is the one i built, the other is downloaded from download.falco.org).

Then i tried to fixup kernelurls to only use rt entries (plus the kbuild one), and... magic! The results are exactly the same:

7ecb5718117b373f6c58148bd1dc2fdf  /tmp/falco_debian_4.19.282-1-amd64_1.ko
7ecb5718117b373f6c58148bd1dc2fdf  falco_debian_4.19.282-1-amd64_1.ko

So, we are building the rt configuration. Finally, i tried to build using only the non-rt kernelurls and:

8421e03822e4dcda2d4065ead5e7760e  /tmp/falco_debian_4.19.282-1-amd64_1.ko
7ecb5718117b373f6c58148bd1dc2fdf  falco_debian_4.19.282-1-amd64_1.ko

They differ! I can share the kmod with you @glpfloss , can you test if you can inject it? If that's the issue, the fix is to "simply" update kernel-crawler to generate multiple entries (one for normal, one for -rt, one for -cloud); that is a bug btw.
This should not even require a Falco patch release :)

falco_debian_4.19.282-1-amd64_1.ko.zip

:pray: let us know! Thank you!

FedeDP commented 1 year ago

Also, can you share your /etc/os-release file?

FedeDP commented 1 year ago

I opened the PR on kernel-crawler to fix this: https://github.com/falcosecurity/kernel-crawler/pull/147. Once you confirm that the kmod i provided is working fine, we can merge the fix and try to rebuilt all the debian drivers (we need one, eg @leogr , with s3 access, to nuke debian drivers from s3 bucket too!)

FedeDP commented 1 year ago

Also, falco-driver-loader patch to properly support cloud and rt debian kernel flavors: https://github.com/falcosecurity/falco/pull/2627. This means that until Falco 0.36 (or possibly 0.35.1, but for now nobody ever asked us about rt and cloud flavors, therefore it's not a big deal to wait), we will only support normal debian kernel flavor, ie: the one you are using.

glpfloss commented 1 year ago

Whoa, thank you @FedeDP , im going to try to answer all your questions

Care to try using ebpf instead?

With epbf, falco-driver-loader ends ok:

* Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/5.0.1%2Bdriver/x86_64/falco_debian_4.19.282-1-amd64_1.o
* Skipping compilation, eBPF probe is already present in /root/.falco/5.0.1+driver/x86_64/falco_debian_4.19.282-1-amd64_1.o
* eBPF probe located in /root/.falco/5.0.1+driver/x86_64/falco_debian_4.19.282-1-amd64_1.o
* Success: eBPF probe symlinked to /root/.falco/falco-bpf.o

But then, falco container fails with this error:

Thu Jun  8 16:57:44 2023: Enabled event sources: syscall
Thu Jun  8 16:57:44 2023: Opening 'syscall' source with BPF probe. BPF probe path: /root/.falco/falco-bpf.o
Thu Jun  8 16:57:44 2023: An error occurred in an event source, forcing termination...
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
Error: BPF probe is compiled for 4.19.0-24-rt-amd64, but running version is 4.19.0-24-amd64

They differ! I can share the kmod with you , can you test if you can inject it?

I can confirm if i inject that module to the falco-driver-loader it works:

* Found a prebuilt falco module at /root/.falco/5.0.1+driver/x86_64/falco_debian_4.19.282-1-amd64_1.ko, loading it
* Success: falco module found and inserted

Falco container is reporting healthy status after the init, so i guess this is good

Also, can you share your /etc/os-release file?

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Please let me know if i forgot something, thank you for your quick support

FedeDP commented 1 year ago

Yay then my patch is ok! Thank you very much for the quick response! We will fix the upstream artifacts (download.falco.org) asap, during next week!

glpfloss commented 1 year ago

Thank you @FedeDP , please if you don't mind, i would appreciate if you can leave a message here when modules are fixed in the bucket

FedeDP commented 1 year ago

Of course I will! Really thank you for helping us spot this issue!

FedeDP commented 1 year ago

News :loudspeaker: We will be releasing a 0.35.1 Falco patch release that will include my fix :rocket: /milestone 0.35.1

Sorry, we are still unable to fixup published artifacts because we are still waiting for kernel-crawler PR to get merged: https://github.com/falcosecurity/kernel-crawler/pull/147

FedeDP commented 1 year ago

I'll reopen this one because the issue is still alive. We need to replace debian artifacts for drivers 5.0.1+driver. /reopen

poiana commented 1 year ago

@FedeDP: Reopened this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/2625#issuecomment-1604355226): >I'll reopen this one because the issue is still alive. We need to replace debian artifacts for drivers 5.0.1+driver. >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
FedeDP commented 1 year ago

Ehy we finally took the time to republish all debian driver artifacts; the issue should now be solved; here is the new link to your driver: https://download.falco.org/driver/5.0.1%2Bdriver/x86_64/falco_debian_4.19.282-1-amd64_1.ko. I checked and it has exactly same shasum as the one i shared with you a couple of weeks ago:

falco_debian_4.19.282-1-amd64_1.ko.zip

FedeDP commented 1 year ago

/close

poiana commented 1 year ago

@FedeDP: Closing this issue.

In response to [this](https://github.com/falcosecurity/falco/issues/2625#issuecomment-1612594260): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.