Can't load Falco kernel module after building with Kmod Kit

[ghost]

I built the Falco kernel module using kmod kit as per this discussion. However, I can't seem to figure out how to install it. Here are the 2 approaches I tried:

• I tried installing it from a bootstrap container by running insmod, but quickly found out that won't work because the bootstrap container doesn't have root access.
• I tried manually putting the kernel module under /.bottlerocket/rootfs/lib/modules/5.10.118/kernel/drivers/falco/ assuming driverdog will load it, but it didn't. So, I tried SSH-ing and running insmod and got this error:

  [  192.413609] falco: loading out-of-tree module taints kernel.
  [  192.418333] falco: module verification failed: signature and/or required key missing - tainting kernel
  [  192.426190] falco: disagrees about version of symbol find_pid_ns
  [  192.430891] falco: Unknown symbol find_pid_ns (err -22)
  [  192.435185] falco: disagrees about version of symbol task_active_pid_ns
  [  192.440105] falco: Unknown symbol task_active_pid_ns (err -22)
  [  192.444655] falco: disagrees about version of symbol fget
  [  192.449041] falco: Unknown symbol fget (err -22)
  [  192.453054] falco: disagrees about version of symbol init_pid_ns
  [  192.457857] falco: Unknown symbol init_pid_ns (err -22)
  [  192.462145] falco: disagrees about version of symbol init_task
  [  192.466739] falco: Unknown symbol init_task (err -22)
  [  192.470937] falco: disagrees about version of symbol __task_pid_nr_ns
  [  192.475868] falco: Unknown symbol __task_pid_nr_ns (err -22)
  [  192.480351] falco: disagrees about version of symbol pid_task
  [  192.484879] falco: Unknown symbol pid_task (err -22)
  [  192.489021] falco: disagrees about version of symbol task_cputime_adjusted
  [  192.494062] falco: Unknown symbol task_cputime_adjusted (err -22)
  [  192.498725] falco: disagrees about version of symbol device_create
  [  192.503458] falco: Unknown symbol device_create (err -22)
  [  192.507837] falco: disagrees about version of symbol device_destroy
  [  192.512595] falco: Unknown symbol device_destroy (err -22)
  [  192.517013] falco: disagrees about version of symbol inode_permission
  [  192.521851] falco: Unknown symbol inode_permission (err -22)
  [  192.526333] falco: disagrees about version of symbol sockfd_lookup
  [  192.531055] falco: Unknown symbol sockfd_lookup (err -22)

Not sure where to go from here, any help will be great!

[arnaldo2792]

Hello @youssufelshall , thanks for reporting this problem. Let me do some debugging first to understand what's happening :+1: .

[ghost]

So, I made sure the kernel sources are being referenced correctly and copied Modules.symvers from the kernel sources to the Falco driver source, and for some reason after building the module it is emptied.

[bcressey]

@arnaldo2792 any luck with the debugging?

@youssufelshall it's not a fix for the kmod kit interaction, but for Falco specifically you can use the eBPF probe driver rather than the kmod, per https://github.com/bottlerocket-os/bottlerocket/issues/862#issuecomment-720122492.

The command has evolved a bit since that time, but this helm command should work with the current chart:

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=ebpf \
  --set collectors.crio.enabled=false \
  --set collectors.docker.enabled=false \
  --set collectors.containerd.socket=/run/dockershim.sock \
  --set podSecurityContext.seLinuxOptions.user=system_u \
  --set podSecurityContext.seLinuxOptions.role=system_r \
  --set podSecurityContext.seLinuxOptions.type=control_t \
  --set podSecurityContext.seLinuxOptions.level=s0-s0:c0.c1023

The SELinux options are there to fix two problems I noticed. The first is that there doesn't seem to be a way to run the driver loader init container as privileged, which it needs to be to modify the host's /usr/src/kernels during the build. The second is that the Falco container attempts to traverse various host paths which are only accessible to privileged containers (because they contain secrets) which generates a large number of AVC denials.

Effectively this runs all the containers in the pod with the label that a privileged container would have, without also giving them the other privileges like capabilities.

You can omit the --set driver.kind=ebpf and that also seems to work, but you have to disable lockdown on the host as well.

[arnaldo2792]

I did debug the problem (I was having a hard time understanding a permissions error, turns out I had lockdown=integrity). I confirmed that I can load the kernel module after I compiled it using what is proposed in #1725.

> insmod /tmp/falco/build/driver/falco.ko
# ...
# From dmesg
Jul 18 18:58:23 ip-192-168-48-109.us-west-2.compute.internal kernel: falco: loading out-of-tree module taints kernel.
Jul 18 18:58:23 ip-192-168-48-109.us-west-2.compute.internal kernel: falco: module verification failed: signature and/or required key missing - tainting kernel
Jul 18 18:58:23 ip-192-168-48-109.us-west-2.compute.internal kernel: falco: driver loading, falco 2.0.0+driver

@youssufelshall, my gut feeling is that you are trying to load the kernel module in a kernel version different than what was used to build the kmod-kit. Could you please share the AMI id and the kmod-kit version you are using?

[ghost]

Turns out the issue was that the Module.symvers that was with the kernel source code was different from the one on the actual node, despite checking the version numbers and making sure everything is right. I just copied Module.symvers from one of the nodes and built the kernel module successfully.