falcosecurity / falco

Cloud Native Runtime Security
https://falco.org
Apache License 2.0
7.16k stars 884 forks source link

Discussion: Updating the default falco-driver-loader image #2489

Closed LucaGuerra closed 11 months ago

LucaGuerra commented 1 year ago

As we know we have three main avenues install the system instrumentation to get live syscall info into Falco:

The first two are the most compatible but require building the drivers or downloading a prebuilt for your kernel. While we have thousands of prebuilts it is sometimes necessary to build your own driver. If you're using a container-based install, this is what the falco-driver-loader image attempts to do.

That image has a noble purpose, to try and build the driver for the live system it is on provided that kernel headers are present, but it's not without issues. For historical reasons, a ton of old compilers are still installed in a relatively old image (Debian Buster) to retain compatibility with older kernels (3.x and below). However, that image is unable to build the driver for newer versions (kernel > 5.0 and perhaps some 4.xx as well). In addition, if you try to run the image through a SCA / vulnerability scanner tool it will alert about maybe hundreds of vulnerabilities which are present in very old compilers and accompanying libs that are necessary for older distributions; for all practical security purposes those are all false positives but they would show up flashing on a security status dashboard and would make compliance experts feel uneasy.

The sad fact is that it's not even possible to build an image that works for everyone. In fact, even driverkit has resorted to using several different images to to this job.

This led me to try and look into the compiler versions we actually use when creating our prebuilt and trying to understand if we want to upgrade the default driver loader image. To learn which compilers we may want to put there I ran a little experiment: took all the current driverkit configs we use and looked at all the image selections. The results are as follows (after removing duplicate kernel versions):

Count Name
----- ----
    1 falcosecurity/driverkit-builder-any-x86_64_gcc10.0.0_gcc9.0.0:latest
 1457 falcosecurity/driverkit-builder-any-x86_64_gcc12.0.0_gcc11.0.0:latest
  399 falcosecurity/driverkit-builder-any-x86_64_gcc8.0.0_gcc6.0.0_gcc5.0.0_gcc4.9.0_gcc4.8.0:latest  <<-- this is the current Falco image
    1 falcosecurity/driverkit-builder-centos-x86_64_gcc4.8.5:latest

Basically, we use the gcc 11-12 image most of the times. In fact, as a rule of thumb, we use the gcc11-12 image for 5.x and 6.x while we use the old image for 4.x and below. So, we're recently picking up many variations of kernel 5.x because it seems like it's the most widely supported at the moment (Ubuntu, Amazon Linux, some recent RHEL and CentOS...) while 4.x versions are not seeing many updates.

I believe it may be time to upgrade the base image to something like this: https://github.com/falcosecurity/driverkit/blob/master/docker/builders/builder-any-x86_64_gcc12.0.0_gcc11.0.0.Dockerfile since building on 5.x and 6.x seems to be becoming more important than the previous versions. If we could also add another gcc version that is compatible with some 4.x I believe it would work even better.

This issue acts as a place of discussion about what we want to do. There may be more ideas, and even more things we want to do for people that cannot use this base image, but I did want to open this discussion.

/kind design

poiana commented 1 year ago

There is not a label identifying the kind of this issue. Please specify it either using /kind <group> or manually from the side menu.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
LucaGuerra commented 1 year ago

I can also confirm that we use GCC 8 for all kernels 4.x, but the compiler version is not enough to be sure about whether a specific image will work with a specific kernel version.

tspearconquest commented 1 year ago

Double confirming the last comment. I tried to build and load the dkms driver from source with gcc8, 9, 10, and 11, and it would not work with any for me on Kernel 5.4 (AKS K8s 1.24 on Ubuntu 18.04), so I went back to the pre-built driver.

LucaGuerra commented 1 year ago

Kernel 5.4 (AKS K8s 1.24 on Ubuntu 18.04)

There was I, thinking that Ubuntu 18.04 was on kernel 4.x

Anyways, yes. Unfortunately it's practically impossible to create an image that is able to build the kmod or classic eBPF probe for every possible kernel version and distribution, and even if we could somehow do it it'd be extremely large in size making it impractical. And yet, we want to distribute a Falco image that is able to build it for some popular Linux distributions to make it easier to run Falco.

FedeDP commented 1 year ago

Well, building the eBPF probe should work with any clang version >=7, therefore we should always be able to build it. The only real issue here is building the kmod, IMHO. I agree with the idea of bumping the falco-driver-loader toolchains to eg: gcc 11/12 and a more recent clang, we should cover most of our users needs (that have shifted from old kernel {3,4}.x to newer {5,6}.x).

Andreagit97 commented 1 year ago

I think we need to tackle this issue ASAP, in the meanwhile we have already collected at least 4 issue regarding this topic :/ :point_up:

FedeDP commented 1 year ago

:+1: from me; open to give any help needed!

tspearconquest commented 1 year ago

Meanwhile, I've upgraded our clusters to 1.25 and 1.26. On AKS, this means we're now on Ubuntu 22.04 which has a kernel that can use the modern bpf probe.

Happy to still test out fixes for the build issues with any of the three options though!

Get Outlook for Androidhttps://aka.ms/AAb9ysg


From: Federico Di Pierro @.> Sent: Thursday, June 15, 2023 4:28:25 AM To: falcosecurity/falco @.> Cc: Thomas Spear @.>; Comment @.> Subject: Re: [falcosecurity/falco] Discussion: Updating the default falco-driver-loader image (Issue #2489)

CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

👍 from me; open to give any help needed!

— Reply to this email directly, view it on GitHubhttps://github.com/falcosecurity/falco/issues/2489#issuecomment-1592696429, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATRTFZ2SWD34TYL4L4N3SXTXLLITTANCNFSM6AAAAAAW3XHZUY. You are receiving this because you commented.Message ID: @.***>