Falco Binary Downloads Improvements

krisnova commented 4 years ago

Describe the bug

After downloading the tar.gz I was unable to successfully load a Falco driver and start the Falco daemon.

How to reproduce it

Download the latest binary package for 0.22.1.

See #1179 for detail on how I discovered this URL

wget https://dl.bintray.com/falcosecurity/bin/x86_64/falco-0.22.1-x86_64.tar.gz
tar -xzf falco-0.22.1-x86_64.tar.gz
cd falco-0.22.1-x86_64
tree .
.
├── etc
│   └── falco
│       ├── falco_rules.local.yaml
│       ├── falco_rules.yaml
│       ├── falco.yaml
│       ├── k8s_audit_rules.yaml
│       ├── rules.available
│       │   └── application_rules.yaml
│       └── rules.d
└── usr
    ├── bin
    │   ├── falco
    │   └── falco-driver-loader
    ├── share
    │   └── falco
    │       └── lua
    │           ├── compiler.lua
    │           ├── lyaml
    │           │   ├── explicit.lua
    │           │   ├── functional.lua
    │           │   └── implicit.lua
    │           ├── lyaml.lua
    │           ├── output.lua
    │           ├── parser.lua
    │           ├── rule_loader.lua
    │           ├── sinsp_rule_utils.lua
    │           └── test.lua
    └── src
        └── falco-a259b4bf49c3330d9ad6c3eed9eb1a31954259a6
            ├── bpf
            │   ├── bpf_helpers.h
            │   ├── filler_helpers.h
            │   ├── fillers.h
            │   ├── Makefile
            │   ├── maps.h
            │   ├── plumbing_helpers.h
            │   ├── probe.c
            │   ├── quirks.h
            │   ├── ring_helpers.h
            │   └── types.h
            ├── dkms.conf
            ├── driver_config.h
            ├── dynamic_params_table.c
            ├── event_table.c
            ├── fillers_table.c
            ├── flags_table.c
            ├── main.c
            ├── Makefile
            ├── ppm_compat_unistd_32.h
            ├── ppm_cputime.c
            ├── ppm_events.c
            ├── ppm_events.h
            ├── ppm_events_public.h
            ├── ppm_fillers.c
            ├── ppm_fillers.h
            ├── ppm_flag_helpers.h
            ├── ppm.h
            ├── ppm_ringbuffer.h
            ├── ppm_syscall.h
            ├── ppm_version.h
            └── syscall_table.c

Expected behaviour

So as an arch Linux user I come to expect certain things from tarballs for packages being downloaded from a projects archive.

Documentation

Usually, there is a README or an INSTALL file I can read to explain the directory structure. If there isn't documentation there is typically a ./configure or an init.sh style script that gives users an intuitive way to start the process.

Kernel Module

Going to usr/bin yields two files falco and falco-probe-loader.

This failed for me.

sudo ./falco-driver-loader 
* Unloading falco-probe, if present
* Running dkms install for falco
./falco-driver-loader: line 147: dkms: command not found
* Running dkms build failed, couldn't find /var/lib/dkms/falco/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 5.4.23.a-1-hardened
Found kernel config at /proc/config.gz
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-a259b4bf49c3330d9ad6c3eed9eb1a31954259a6-x86_64-5.4.23.a-1-hardened-22b65324f3dd66a0bcb8f85f84c06746.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community

and I strongly disagree with what that script was trying to do.

I do not think we should ever attempt to download pre-compiled modules without validating them somehow. Furthermore I do not think we should be trying to download the modules from a Sysdig managed S3 account. This needs be fixed immediately.

If you look at openzfs you can see how they manage downloading a system that requires a userspace component and a kernel module. I would very much like it if Falco followed this paradigm. See latest release and try to build from source for an example. We can keep the pre-built modules so long that we are validating them, but if the script fails.

sudo ./falco-driver-loader 
* Unloading falco-probe, if present
* Running dkms install for falco
./falco-driver-loader: line 147: dkms: command not found
* Running dkms build failed, couldn't find /var/lib/dkms/falco/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 5.4.23.a-1-hardened
Found kernel config at /proc/config.gz
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-a259b4bf49c3330d9ad6c3eed9eb1a31954259a6-x86_64-5.4.23.a-1-hardened-22b65324f3dd66a0bcb8f85f84c06746.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community

we need to give users a clear next-step. Which is:

The source to build the kernel module is in usr/src/falco-*/ and can be compiled. Then a user can run make install and we use depmod to install the falco_probe.ko file and it will persist on reboot.

Userspace program

The Falco binary has a linked dependency on having libyaml installed.

sudo ./falco
./falco: error while loading shared libraries: libyaml-0.so.2: cannot open shared object file: No such file or directory

Also on lua

[novix@novix-alice]: ~/src/falco-0.22.1-x86_64/usr/bin>$ sudo ./falco
Sat Apr 25 13:21:06 2020: Runtime error: Could not find Falco Lua entrypoint (tried /usr/share/falco/lua/rule_loader.lua, /source/falco/userspace/engine/lua/rule_loader.lua). Exiting.

We should be cleaning this up and have our Makefiles check for these types of concerns and give clear error messages for next steps.

Screenshots

Environment

[novix@novix-alice]: ~/src/falco-0.22.1-x86_64/usr/bin>$ uname -a Linux novix-alice 5.4.23.a-1-hardened #1 SMP PREEMPT Sun, 01 Mar 2020 18:23:23 +0000 x86_64 GNU/Linux [novix@novix-alice]: ~/src/falco-0.22.1-x86_64/usr/bin>$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz Stepping: 2 CPU MHz: 1200.190 CPU max MHz: 3300.0000 CPU min MHz: 1200.0000 BogoMIPS: 5001.19 Virtualization: VT-x L1d cache: 768 KiB L1i cache: 768 KiB L2 cache: 6 MiB L3 cache: 60 MiB NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47 Vulnerability Itlb multihit: KVM: Mitigation: Split huge pages Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1 gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand l ahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d [novix@novix-alice]: ~/src/falco-0.22.1-x86_64/usr/bin>$

Falco version:
System info:
Cloud provider or hardware configuration:
OS:
Kernel:
Installation method:

Additional context

krisnova commented 4 years ago

Success:

Clear documentation, error messages, and always providing the user a next-step when building form the binary packages.

Right now even as a Falco maintainer I was a bit confused with what was next.

krisnova commented 4 years ago

/kind user-exerpience

leodido commented 4 years ago

The S3 link has been fixed and the fix will be released in 0.23.0

The fix is already present on development releases (the falco-driver-loader shipped into falcosecurity/falco:master already have this fix, for example).

Totally agree that falco-driver-loader should give more clear messages and steps to the users.

Also, it should check in advance the presence of dkms.

I do not think we should ever attempt to download pre-compiled modules without validating them somehow.

What do you mean with validating in this specific case? Could you please make an example of validation?

Do you mean download it only if we are sure a prebuilt for the specific kernel exists?

If this is what you mean we can implement it easily and cleanly with a HEAD request.

krisnova commented 4 years ago

We should validate the md5 or sha just like we do with all of our downloads. Just to ensure it's what we expect it to be.

Check out this tarball I just made that shows what I am thinking for user experience for downloading the tar.gz's from bintray

http://nivenly.com/falco/falco-0.22.1-suggest.tar.gz

krisnova commented 4 years ago

Basically - it should be very clear and obvious that the user needs to make a choice to either build something or download it or try to download it from the internet.

I think we should remove any assumptions or "magic".

Do you want to TRY to download a kernel module from the internet or build it? Pick one.

krisnova commented 4 years ago

I would suggest downloading the ZFS 8.0.3 release and check out the user experience here with loading the module and configuring systemd.

https://github.com/openzfs/zfs/releases/tag/zfs-0.8.3

The architecture of ZFS is almost identical to that of Falco - and their tarballs are very nice :nerd_face:

krisnova commented 4 years ago

If someone can show me where I could make changes to the build to make a PR I will open one up.

leogr commented 4 years ago

I believe there are distinct issues here. I try to summarize my thoughts:

falco-driver-loader
- does not give clear messages to the user, UX issue that should be fixed
- should verify downloads prior to using them, I can't disagree, although the best solution might be this?
- should allow the users to skip the build or the download step IMO, but its default behavior should remain as is now because it seems to me that's intended for automation tasks (eg. falco/driverloader image that can be used as initcontainer) and not really for end-users
the binary package (the official link is present within the artifact table into the Falco's README)
- has no documentation, and I strongly agree we have to explain how to use it
- does not seem to me that is intended for building Falco from source, although it contains a special case to just build the driver on-the-fly; in fact, it does not work unless you install it (eg. copy files to the right location into your filesystem) - correct me if my understanding is wrong!
- that being said, why should we endorse another "building from source" strategy if we have one yet?

Anyway, I believe we can do a lot of improvements here, and I would like to help with this, let me know!

krisnova commented 4 years ago

This is coming along nicely with the recent changes.

Thanks for your help @leogr

Is binary still the best name for this?

leogr commented 4 years ago

As per this proposal, I think binary is the best name because it is NOT intended to build Falco from source.

Also note that:

although it's a binary package, it includes driver sources that are just used by the script to build it on-the-fly (consider this as special case, that we need for technical reasons)
- since 0.23.0, falco-driver-loader was modularized and you can type falco-driver-loader --help now.
- the online documentation has been updated, see here.

That being said, we still miss INSTALL and Makefile files inside the package. So we should keep this issue open until it's fixed.

Also, we want to add source packages too, as per this proposal, but it's a different issue IMHO.

krisnova commented 4 years ago

Related to #1265 if we are making changes to the packages

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Issues labeled "cncf", "roadmap" and "help wanted" will not be automatically closed. Please refer to a maintainer to get such label added if you think this should be kept open.

falcosecurity / falco