Debian 11 kernel version issue

[serhatcalik]

Describe the bug

I was trying to use falco on my kubernetes cluster with drivers, on the init container it's trying to download the kernel module but it's trying to download 5.10.0-20-amd64.o file from download.falco.org domain, on debian it's showing 5.10.0-20 kernel version but it's actually 5.10.158-2 but script is getting uname -r's response so it's trying to download wrong file.

How to reproduce it

On any debian 11 machine with k8s cluster try to install falco.

Expected behaviour

It needs to download 5.10.0-20-amd64.o file from download server.

Screenshots

Environment

• Falco version: 0.33.1

• System info: Wed Jan 25 10:37:56 2023: Falco version: 0.33.1 (x86_64)
 Wed Jan 25 10:37:56 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
 Wed Jan 25 10:37:56 2023: Loading rules from file /etc/falco/falco_rules.yaml
 Wed Jan 25 10:37:56 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
 null

• Cloud provider or hardware configuration:

• OS:

  PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"

  NAME="Debian GNU/Linux"

  VERSION_ID="11"

  VERSION="11 (bullseye)"

  VERSION_CODENAME=bullseye
ID=debian
  
HOME_URL="[https://www.debian.org/"](https://www.debian.org/%22)

  
SUPPORT_URL="[https://www.debian.org/support"](https://www.debian.org/support%22)

  
BUG_REPORT_URL="[https://bugs.debian.org/"](https://bugs.debian.org/%22)

  


• Kernel:

  Linux ranchm0 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux

• Installation method: Kubernetes

[Andreagit97]

Uhm this is more an issue with uname, it should report the running kernel version but in this case, it seems to report the wrong one :/ Have you rebooted the machine? Are you sure that you are running the new kernel version? maybe it is just installed but not running

[serhatcalik]

Uhm this is more an issue with uname, it should report the running kernel version but in this case, it seems to report the wrong one :/ Have you rebooted the machine? Are you sure that you are running the new kernel version? maybe it is just installed but not running

I tried but still same output from uname -r :(

[Andreagit97]

Uhm really strange :/ Could you provide the output of the following commands?

• uname -a
• hostnamectl
• cat /proc/version
• sudo dmesg | grep Linux

[FedeDP]

I think i found something: https://lists.debian.org/debian-user/2017/03/msg00485.html

It seems like you are running kernel 5.10.158-2 that is compatible with the kernel ABI used in Debian kernel package 5.10.0-20-amd64.

We did never encountered such an uname -a output :/ it seems a bit crazy to me that uname -r is reporting the "kernel ABI package" and not the real running version, if i understood that correctly.

[serhatcalik]

Uhm really strange :/ Could you provide the output of the following commands?

• uname -a
• hostnamectl
• cat /proc/version
• sudo dmesg | grep Linux

Updated to the newer kernel for testing but still same

[serhatcalik]

I think i found something: https://lists.debian.org/debian-user/2017/03/msg00485.html

It seems like you are running kernel 5.10.158-2 that is compatible with the kernel ABI used in Debian kernel package 5.10.0-20-amd64.

We did never encountered such an uname -a output :/ it seems a bit crazy to me that uname -r is reporting the "kernel ABI package" and not the real running version, if i understood that correctly.

Maybe for debian 11 we can use symbolic link kernels on falco's download page. It's really weird :(

[FedeDP]

@serhatcalik yep, the issue is:

• every piece of software uses uname -r
• on your specific debian installation, uname -r does not return the real running kernel release, but instead returns the Debian kernel package providing the compatible kernel ABI

I am not sure how to tackle it :(

[FedeDP]

Maybe for debian 11 we can use symbolic link kernels on falco's download page. It's really weird :(

We might want to do that, but we haven't got any way to link a kernel release to its ABI compatible Debian kernel package version :) I think we might want to instead parse the kernel version (uname -v) string, and if it contains a semver, we could try both uname -r (so called kernel release) and the semver contained inside the kernel version.

Given we are really near Falco 0.34.0, i will put this for 0.35.0 (perhaps it could get merged in a 0.34.1 patch release, if needed). /milestone 0.35.0

I will try to fix it and perhaps i can publish a falco-driver-loader image on my own dockerhub repo, if you are willing to test it :) NOTE: we still haven't got any 5.10.162-1 kernel build for debian, therefore you must wait for next week kernel-crawler sync before actually finding drivers for your new kernel!

[serhatcalik]

Maybe for debian 11 we can use symbolic link kernels on falco's download page. It's really weird :(

We might want to do that, but we haven't got any way to link a kernel release to its ABI compatible Debian kernel package version :) I think we might want to instead parse the kernel version (uname -v) string, and if it contains a semver, we could try both uname -r (so called kernel release) and the semver contained inside the kernel version.

Given we are really near Falco 0.34.0, i will put this for 0.35.0 (perhaps it could get merged in a 0.34.1 patch release, if needed). /milestone 0.35.0

I will try to fix it and perhaps i can publish a falco-driver-loader image on my own dockerhub repo, if you are willing to test it :) NOTE: we still haven't got any 5.10.162-1 kernel build for debian, therefore you must wait for next week kernel-crawler sync before actually finding drivers for your new kernel!

I would be glad if you can share your image :)

[FedeDP]

I just pushed the 0.33.1_debianfix tag on my own dockerhub: https://hub.docker.com/layers/fededp/falco-driver-loader/0.33.1_debianfix/images/sha256-06db31d8d31bcd80b451329348f63c65f37249b8c9d95032cccd700849919f57?context=repo

As i said, we just need to test that the kernel release it tries to find is the correct one now, but the prebuilt driver won't be available :) The patch is the one in #2377

[serhatcalik]

I just pushed the 0.33.1_debianfix tag on my own dockerhub: https://hub.docker.com/layers/fededp/falco-driver-loader/0.33.1_debianfix/images/sha256-06db31d8d31bcd80b451329348f63c65f37249b8c9d95032cccd700849919f57?context=repo

As i said, we just need to test that the kernel release it tries to find is the correct one now, but the prebuilt driver won't be available :) The patch is the one in #2377

Thank you so much :)

[serhatcalik]

@FedeDP Hey I have bad news

• Trying to download a prebuilt falco module from https://download.falco.org/driver/3.0.1%2Bdriver/x86_64/falco_debian_5.10.158-2_1.ko but file is;

https://download.falco.org/driver/3.0.1%2Bdriver/x86_64/falco_debian_5.10.158-2-amd64_1.ko

https://download.falco.org/driver/3.0.1%2Bdriver/x86_64/falco_debian_5.10.158-2-amd64_1.o

We just need to add arch :/

[FedeDP]

Uh you are right! I forgot about it! Will ping you back once i've got an updated image :)

[FedeDP]

I pushed a new version of my falco-driver-loader, care to try? Thank you!

[serhatcalik]

Falco-driver-loader works fine now but the funny part is kernel is getting invalid module format error I'm starting to hate Debian 11 so much

[FedeDP]

Wow, so in the end the 5.10.158 is not your running kernel :/ It seems so, at least.

Updated to the newer kernel for testing but still same

Didn't you upgrade to 5.10.162 though?

[serhatcalik]

Downgraded kernel to the 5.10.158 for testing faster but I can upgrade it.

[FedeDP]

Oh i see, thank you! It makes sense. I don't get the error then...Can you share the dmesg output after trying to insmod the driver?

[serhatcalik]

disagrees about version of symbol module_layout This is the output of falco from dmesg

[FedeDP]

Wow; it seems the prebuilt module is wrong :/ From SO:

This indicates you have compiled the module against a different version of the kernel than is running. Note that even if the running kernel and kernel source have the same numerical value (e.g. both are 2.6.31-20-server), if the two use different configuration options, you may see this error.

Can you test with eBPF by chance?

[serhatcalik]

I tested eBPF and it was not created for debian kernel :(

An error occurred in an event source, forcing termination...

Error: BPF probe is compiled for 5.10.0-20-rt-amd64, but running version is 5.10.0-20-amd64

[FedeDP]

5.10.0-20-rt-amd64, but running version is 5.10.0-20-amd64

O.O Can you share output of ls /lib/modules/ ?

[serhatcalik]

5.10.0-20-amd64 5.10.0-21-amd64 But I'm using 20 not 21 and it's not rt for sure :D

[FedeDP]

Hi! Sorry for the long time with no answer. I am really not sure how to proceed from here :/ Your only chance is to try to let falco-driver-loader building a driver for you, instead of downloading a prebuilt module. Can you try? You should attach to a node and run falco-driver-loader --compile. Then, on the same node, just try to start falco and see if it works. Thank you!

[FedeDP]

There have been more and more issues about this:

• https://github.com/falcosecurity/test-infra/pull/1097
• https://github.com/falcosecurity/kernel-crawler/issues/114

I think we need a fix. @serhatcalik were you able to make it work by chance (ie: after upgrading the kernel perhaps)?

[serhatcalik]

I stopped using debian in my home cluster started using RHEL. But I can reproduce it for to try fixing it on debian we can work on it together :)

[rao-hassan-1]

I am also affected by this bug. Please provide a resolution for this. I have opened a pull request for the issue I was seeing but that won't fix the problem

[FedeDP]

@serhatcalik that was the correct move indeed :rofl: I will test with the help of @rao-hassan-1 :)

[FedeDP]

I just pushed a new image: fededp/falco-driver-loader:0.34.1_fixed. It is the same as falcosecurity/falco-driver-loader:0.34.1 image but with #2377 fix applied.