Absorb features from thought-machine/falco-probes

[VJftw]

👋 , I posted on the Kubernetes #falco Slack channel about our thought-machine/falco-probes project which automates the building and mirroring of eBPF kernel probes for Falco. We'd love to share and contribute some of our solutions to driverkit which help anyone else that has similar needs to us :smile:.

Motivation

We desire to deploy Falco without compiling kernel eBPF probes at runtime as we have a great interest in running Falco with the least amount of privileges possible and in a more immutable fashion.

For us to do this, we download probes that our Kubernetes clusters might need at build-time and copy them into a docker.io/falcosecurity/falco:0.29.1 based image.

During our attempt to build this image, we came across the following hurdles:

1. eBPF probes for amazonlinux2 and cos were missing from download.falco.org/driver.
2. Falco drivers are re-uploaded every build (see https://prow.falco.org/view/s3/falco-prow-logs/logs/build-drivers-amazonlinux2-periodic/1430002167883763712) which risks file hash changes. This meant that we couldn't rely on the download.falco.org/driver repository as a driver source because we perform hash verification on all external files.
3. New kernels for an operating system are supported via a PR process which adds a maintenance burden and human dependency.

It was clear that we needed to compile eBPF probes ourselves, where we encountered the following extra hurdle with driverkit:

4. cos is not supported and doesn't appear trivial to support for the accepted kernelrelease + kernelversion inputs as Kernel headers are provided per cos Build ID, i.e. Translating a kernelrelease + kernelversion to a Build ID programmatically isn't possible as no mapping appears to exist.

Feature

thought-machine/falco-probes is our solution to those 4 hurdles. I've summarised them below:

1. eBPF probes are built and mirrored.
2. Only mirror probes that aren't already mirrored.
3. To solve this, we endeavoured to dynamically build eBPF probes for all possible kernels associated with an operating system. This is possible if we can enumerate them, e.g.
   • amazonlinux2: Run docker run --rm amazonlinux:2 yum --showduplicates list kernel-devel. This is similar for all operating systems which include a package manager for distributing kernel updates.
   • cos: Is a bit trickier as there is no package manager. However, we can run gsutil ls -r gs://cos-tools/ | grep "kernel-headers.tgz$" | cut -f4 -d/ to list all of the Build IDs of cos. This isn't specifically a list of kernels but is an enumerable list that we can trust to build kernel probes from. There will be duplicated probes as different BUILD IDs will likely share kernels.
4. With our solution to 3 in mind, we decided to use the items from the lists as inputs to building a probe. e.g. amazonlinux2 + 4.14.77-86.82.amzn2 or cos + 16108.470.11. We refer to these items as Kernel Package Names.

Detailed discovery and implementation of these solutions are documented in thought-machine/falco-probes/docs/.

In terms of supporting these features within driverkit and test-infra, I think:

1. Is trivial to do: Add output.probe field to each of the **/*.yaml files in https://github.com/falcosecurity/test-infra/tree/master/driverkit/config.
2. Is trivial to do: Check in the S3 bucket which probes already exist and only build + upload the missing ones.
3. and 4. Are not trivial to do: Our solution uses a different set of inputs to the build process than driverkit currently does (kernelrelease + kernelversion vs. kernel package name). Maybe we could support the kernel package name as a different input that obtains kernel sources + headers the same way that thought-machine/falco-probes does? If so, we could easily generate the driverkit YAML input files too with the kernel package name set.

Any other ideas on this? How does this sound to you? :smile:

Alternatives

Described in Motivation 🤞. For solution 3). we could solve this for operating systems with package managers in driverkit by having a tool that generates the kernelrelease + kernelversion inputs by enumerating the available kernel packages.

Additional context

Hopefully, we've covered the context in the Motiviation, but please ask us for more if you're missing any 😄.

[leogr]

/cc @leodido @fntlnz

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[Issif]

Hello @VJftw,

Thank you for this really valuable issue. We love to discuss about in a Community Call:

Falco Weekly Community call every Wednesday (4:00pm to 5:00pm (UTC) (Zoom Details: 
https://zoom.us/my/cncffalcoproject )

or in our Slack Channel https://kubernetes.slack.com #falco

Driverkit and all ways to improve the UX with modules/probes are part of our current topics.

[Issif]

/remove-lifecycle stale

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[leogr]

/remove-lifecycle stale /help

[poiana]

@leogr: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/falcosecurity/driverkit/issues/103): >/remove-lifecycle stale >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[leogr]

cc @FedeDP @dwindsor wdyt?

[dwindsor]

cc @FedeDP @dwindsor wdyt?

I feel like we can close this. kernel-crawler integration solves a lot of the issues mentioned here.