[BUG] Non root user throws error "Falco failed to install. Check Kernel Headers are installed on the Host Systems(K8s)."

daniel-wilmes commented 1 year ago

Describe the bug Running the non_root_user test fails because of falco not being installed. Falco driver has this error on deployment:

Looking for a falco module locally (kernel 5.15.0-67-generic)
Filename 'falco_ubuntu-generic_5.15.0-67-generic_74.ko' is composed of:
driver name: falco
target identifier: ubuntu-generic
kernel release: 5.15.0-67-generic
kernel version: 74
Found a prebuilt falco module at /root/.falco/4.0.0+driver/x86_64/falco_ubuntu-generic_5.15.0-67-generic_74.ko, loading it insmod: ERROR: could not insert module /root/.falco/4.0.0+driver/x86_64/falco_ubuntu-generic_5.15.0-67-generic_74.ko: Operation not permitted

I was able to run this test prior to the current cnf test suite release.

We are running this in kind for our environment.

To Reproduce Steps to reproduce the behavior:

Install Falco on host machine as suggest with the logs (https://falco.org/docs/getting-started/installation/)
Run the non_root_user test
⏭️ SKIPPED: Skipping non_root_user: Falco failed to install. Check Kernel Headers are installed on the Host Systems(K8s). Expected behavior Falco is installed and the test is run for non root user

Screenshots

Device (please complete the following information):

OS Linux
Distro Ubuntu
Version 20.04
Architecture x86

Here are the logs when running the test suite.

non_root_user.log

HashNuke commented 1 year ago

Attempting to reproduce by installing Falco on host machine. Tried modern ebpf for Falco. Test was skipped due to falco k8s daemonset not installing as expected.

From trying out a falco install, I see that the Falco install option chosen was kmod. Will attempt to reproduce issue again with the kmod falco install option.

HashNuke commented 1 year ago

I tried installing falco with kmod, ebpf and modern ebpf on latest Ubuntu LTS. The the falco k8s app installed by the testsuite fails to complete (and therefore affecting the test). Will look into the issue.

HashNuke commented 1 year ago

Updates to the testsuite for this ticket are in the bug/1768 branch. Please read below for details on some changes.

[Issue-1] Wrong option passed to enable the ebpf driver

Falco helm install wasn't locked to a specific chart version. The option passed to the helm command was outdated and the chart was using the kernel module by default.

Solution: I've locked it to use Falco helm chart version 3.1.5 in the working branch. And updated the option to set the driver to ebpf.

[Issue-2] Limited availability of Falco's prebuilt drivers

Falco provides prebuilt drivers only for the 3 recent driver versions (reference)
Availability of prebuilt drivers can be checked here.

From my own tests on Ubuntu (22.04, 22.10) and Debian 11, I found it hard to pinpoint a specific kernel version for which Falco prebuilt drivers are available. Seems like building a custom driver is straightforward.

How to detect driver issues

When Falco cannot find the prebuilt driver, the logs for the Falco pods would look like below.

Also notice that it looks for the kernel headers in lib/modules to attempt to build a driver. If the kernel headers on the nodes are available and /lib/modules on host is mounted as /host/lib/modules on the falco pods, then Falco will also try to build the driver for the kernel.

Solution

In the absence of prebuilt drivers, following are the available options:

Allow Falco helm chart's init containers to build the drivers using the kernel headers.
Compile a driver for the kernel version using Falco's driverkit project. This driver can then be made available to the Falco pods using volume mounts.

These drivers may be a kernel module or ebpf probe.

Using these solutions may require passing custom Helm values for the Falco helm chart installation. I've added the FALCO_HELM_OPTS env var to help with this.

Here's how I've been using this env var:

Built the driver for my kernel as an ebpf probe.
Add Falco helm chart overrides in falco-values.yml
Export FALCO_HELM_OPTS to run the testsuite like below.

FALCO_HELM_OPTS="-f falco-values.yml" ./cnf-testsuite non_root_user

I've been referring to Falco chart's values.yml to construct my yaml file for Falco install. Below is an example to mount the custom ebpf probe I built for my kernel.

Note: By default the only option the testsuite passes to Falco's helm chart install is driver.kind=ebpf. When FALCO_HELM_OPTS is passed, no other options are passed by the testsuite. This is to allow users to use the kernel module driver if required.

driver:
  enabled: true
  kind: ebpf
  ebpf:
    path: "/falco-driver/falco-ubuntu.o"
mounts:
  volumes:
    - name: "driver-fs"
      hostPath:
        path: "/falco-driver"

  volumeMounts:
    - mountPath: "/falco-driver"
      name: driver-fs

agentpoyo commented 1 year ago

Acceptance Criteria

[x] Setup or install k8s cluster for testing.
[x] Follow instructions for installing ebpf falco module on host system outline by hashnuke
[x] Checkout main branch and setup cnf-testsuite from source install instructions
[x] Setup CNF like coredns from samples directory
[x] Setup the falco-values.yml file
[x] Now when I run FALCO_HELM_OPTS="-f falco-values.yml" ./cnf-testsuite non_root_user, I should see this pass and not skip due to headers not installed to support Falco.

lixuna commented 9 months ago

@daniel-wilmes Is this issue still occurring? If so, please share additional information to help with debugging.

lixuna commented 8 months ago

@daniel-wilmes Is this issue still occurring? If so, please share additional information to help with debugging.

HashNuke commented 8 months ago

@lixuna We discussed closing this ticket on the call and opening an enhancement ticket to discuss the test's implementation (or replacement if required) due to the effort required to configure the test to be run.

I'll create another ticket for the enhancement/discussion.

cnti-testcatalog / testsuite