Issues with Falco on ARM7

[vpetersson]

First, good job with Falco. It's an impressive tool to say the least.

I've been trying to get Falco to work on ARM7 this weekend, and made good progress. After a bit of effort (having to recompile the kernel and build Falco from source), I was able to get it to launch.

# falco
Sun Apr 28 09:29:09 2019: Falco initialized with configuration file /home/pi/src/falco/falco.yaml
Sun Apr 28 09:29:09 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Sun Apr 28 09:29:12 2019: Loading rules from file /etc/falco/falco_rules.local.yaml:
Sun Apr 28 09:29:13 2019: Runtime error: error creating the process list. Make sure you have root credentials.. Exiting.

In the above snippet, I'm running as root, so the error is of course misleading.

Looking closer at the logs, I got these errors:

Apr 28 09:29:09 arm-lab falco[2801]: Falco initialized with configuration file /home/pi/src/falco/falco.yaml
Apr 28 09:29:09 arm-lab falco[2801]: Loading rules from file /etc/falco/falco_rules.yaml:
Apr 28 09:29:12 arm-lab falco[2801]: Loading rules from file /etc/falco/falco_rules.local.yaml:
Apr 28 09:29:12 arm-lab kernel: falco_probe: adding new consumer 2f54258e
Apr 28 09:29:12 arm-lab kernel: falco_probe: initializing ring buffer for CPU 0
Apr 28 09:29:12 arm-lab kernel: falco_probe: CPU buffer initialized, size=8388608
Apr 28 09:29:12 arm-lab kernel: falco_probe: initializing ring buffer for CPU 1
Apr 28 09:29:12 arm-lab kernel: falco_probe: CPU buffer initialized, size=8388608
Apr 28 09:29:12 arm-lab kernel: falco_probe: initializing ring buffer for CPU 2
Apr 28 09:29:12 arm-lab kernel: falco_probe: CPU buffer initialized, size=8388608
Apr 28 09:29:12 arm-lab kernel: falco_probe: initializing ring buffer for CPU 3
Apr 28 09:29:13 arm-lab kernel: falco_probe: CPU buffer initialized, size=8388608
Apr 28 09:29:13 arm-lab kernel: falco_probe: starting capture
Apr 28 09:29:13 arm-lab kernel: falco_probe: deallocating consumer 2f54258e
Apr 28 09:29:13 arm-lab kernel: falco_probe: no more consumers, stopping capture
Apr 28 09:29:13 arm-lab kernel: falco_probe: adding new consumer 2f54258e
Apr 28 09:29:13 arm-lab kernel: falco_probe: initializing ring buffer for CPU 0
Apr 28 09:29:13 arm-lab kernel: falco_probe: CPU buffer initialized, size=8388608
Apr 28 09:29:13 arm-lab kernel: falco_probe: initializing ring buffer for CPU 1
Apr 28 09:29:13 arm-lab kernel: falco_probe: CPU buffer initialized, size=8388608
Apr 28 09:29:13 arm-lab kernel: falco_probe: initializing ring buffer for CPU 2
Apr 28 09:29:13 arm-lab kernel: falco_probe: CPU buffer initialized, size=8388608
Apr 28 09:29:13 arm-lab kernel: falco_probe: initializing ring buffer for CPU 3
Apr 28 09:29:13 arm-lab kernel: falco_probe: CPU buffer initialized, size=8388608
Apr 28 09:29:13 arm-lab kernel: falco_probe: starting capture
Apr 28 09:29:13 arm-lab kernel: falco_probe: deallocating consumer 2f54258e
Apr 28 09:29:13 arm-lab kernel: falco_probe: no more consumers, stopping capture
Apr 28 09:29:13 arm-lab falco[2801]: Runtime error: error creating the process list. Make sure you have root credentials.. Exiting.

This was all tested with the kernel 4.19.36-v7+.

I also tried to remove the k8s ruleset, which is irrelevant for my use case.

Any pointers would be appreciated.

[mfdii]

good to see you @vpetersson and thanks for the kind words!

Since you say you recently compiled the kernel, can you verify that CONFIG_AUDITSYSCALL is enabled. I think this might be related to a bug in sysdig (https://github.com/draios/sysdig/issues/1290).

Also, as an aside, since you're on a newer kernel you can use the ebpf probe. It can be compiled from the bpf directory which is where the kernel module code lives.

[vpetersson]

Hi @mfdii - good to hear from you too.

I checked, and the default kernel does not seem to have CONFIG_AUDITSYSCALL enabled. Also, I wanted to use eBPF regardless, so I went down that path instead. However, I did run into some issues.

Default, the system comes with clang 3.8.1, which appears to be rather ancient by now. This generated the following just FYI:

root@arm-lab:/usr/src/falco-0.1.1dev/bpf# make
make -C /lib/modules/4.19.36-v7+/build M=$PWD
make[1]: Entering directory '/mnt/src/linux'
clang -I./arch/arm/include -I./arch/arm/include/generated  -I./include -I./arch/arm/include/uapi -I./arch/arm/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h \
        -D__KERNEL__ -mlittle-endian   \
         \
        -D__KERNEL__ \
        -D__BPF_TRACING__ \
        -Wno-gnu-variable-sized-type-not-at-end \
        -Wno-address-of-packed-member \
        -fno-jump-tables \
        -Wno-tautological-compare \
        -O2 -g -emit-llvm -c /usr/src/falco-0.1.1dev/bpf/probe.c -o /usr/src/falco-0.1.1dev/bpf/probe.ll
clang: error: unknown argument: '-fno-jump-tables'
/usr/src/falco-0.1.1dev/bpf/Makefile:33: recipe for target '/usr/src/falco-0.1.1dev/bpf/probe.o' failed
make[2]: *** [/usr/src/falco-0.1.1dev/bpf/probe.o] Error 1
Makefile:1525: recipe for target '_module_/usr/src/falco-0.1.1dev/bpf' failed
make[1]: *** [_module_/usr/src/falco-0.1.1dev/bpf] Error 2
make[1]: Leaving directory '/mnt/src/linux'
Makefile:18: recipe for target 'all' failed
make: *** [all] Error 2

After upgrading to clang 8.0.0, I did make some progress. However, it eventually choked too unfortunately. The full compile snippet can be found here.

Finally, i gave clang 7.0.1 a shot, and unfortunately it failed to compile too (snippet).

Unfortunately, I'm not too familiar with the this domain space, so I'm not sure exactly how to progress from here. In the meantime, I guess i will play with a recompiled kernel.

[mfdii]

Well, if you recompile your 4.19 kernel with CONFIG_AUDITSYSCALL enabled you should be able to get Falco to start, or at least get further down the path. I know sysdig has been built for ARM in the past, so the kernel module should work. The other option would be to submit a PR that doesn't cause this function to return SCAP_FAILURE if the loginuid file doesn't exist for a process.

The eBPF code is newer and I don't think it's ever been tested/ported to ARM so there's no guarantee it would work. @gianlucaborello might have some suggestions or advice on the feasibility of getting it to work on ARM.

[vpetersson]

Bingo! After a bit of tweaking with the kernel, and lots of time waiting on the compile, I was able to get it to load with the kernel module:

pi@arm-lab:/usr/src/falco-0.1.1dev $ sudo falco
Tue Apr 30 11:06:53 2019: Falco initialized with configuration file /home/pi/src/falco/falco.yaml
Tue Apr 30 11:06:53 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Tue Apr 30 11:06:57 2019: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue Apr 30 11:06:57 2019: Starting internal webserver, listening on port 8765
Tue Apr 30 11:07:09 2019: Falco internal: syscall event drop. 1 system calls dropped in last second.11:07:09.611781161: Critical Falco internal: syscall event drop. 1 system calls dropped in last second.(ebpf_enabled=0 n_drops=1 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=1 n_evts=1993)
Tue Apr 30 11:07:11 2019: Falco internal: syscall event drop. 7 system calls dropped in last second.11:07:11.762391581: Critical Falco internal: syscall event drop. 7 system calls dropped in last second.(ebpf_enabled=0 n_drops=7 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=7 n_evts=4437)
Tue Apr 30 11:07:12 2019: Falco internal: syscall event drop. 78 system calls dropped in last second.11:07:12.779751039: Critical Falco internal: syscall event drop. 78 system calls dropped in last second.(ebpf_enabled=0 n_drops=78 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=78 n_evts=11483)
Tue Apr 30 11:07:14 2019: Falco internal: syscall event drop. 2 system calls dropped in last second.11:07:14.455159681: Critical Falco internal: syscall event drop. 2 system calls dropped in last second.(ebpf_enabled=0 n_drops=2 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=2 n_evts=2295)
Tue Apr 30 11:07:16 2019: Falco internal: syscall event drop. 2 system calls dropped in last second.11:07:16.470473096: Critical Falco internal: syscall event drop. 2 system calls dropped in last second.(ebpf_enabled=0 n_drops=2 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=2 n_evts=1162)
Tue Apr 30 11:07:24 2019: Falco internal: syscall event drop. 2 system calls dropped in last second.11:07:24.864702758: Critical Falco internal: syscall event drop. 2 system calls dropped in last second.(ebpf_enabled=0 n_drops=2 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=2 n_evts=3734)
Tue Apr 30 11:07:25 2019: Falco internal: syscall event drop. 2 system calls dropped in last second.11:07:25.942764235: Critical Falco internal: syscall event drop. 2 system calls dropped in last second.(ebpf_enabled=0 n_drops=2 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=2 n_evts=1480)
Tue Apr 30 11:07:27 2019: Falco internal: syscall event drop. 2 system calls dropped in last second.11:07:27.369571606: Critical Falco internal: syscall event drop. 2 system calls dropped in last second.(ebpf_enabled=0 n_drops=2 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=2 n_evts=1347)
Tue Apr 30 11:07:28 2019: Falco internal: syscall event drop. 2 system calls dropped in last second.11:07:28.585381720: Critical Falco internal: syscall event drop. 2 system calls dropped in last second.(ebpf_enabled=0 n_drops=2 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=2 n_evts=1497)
Tue Apr 30 11:07:33 2019: Falco internal: syscall event drop. 3 system calls dropped in last second.11:07:33.473028583: Critical Falco internal: syscall event drop. 3 system calls dropped in last second.(ebpf_enabled=0 n_drops=3 n_drops_buffer=0 n_drops_bug=0 n_drops_pf=3 n_evts=4667)

Not sure how relevant/critical this error is.

[mfdii]

It's not critical per se as you're only dropping a small percentage of calls, but it is something we are looking at reducing the occurrence of. We've released 0.15.0 if you want to build from that source: https://github.com/falcosecurity/falco/releases/tag/0.15.0

[mfdii]

BTW you can control if these drops are logged or sent as alerts. If you just want logging of drops, remove - alert from the config file

[vpetersson]

Thanks! Compiling now. Will let you know tomorrow when it's done ;)

[fntlnz]

@vpetersson do you have any feedback on this ? 👼

[vpetersson]

@fntlnz Thanks for checking in. I'm still struggling with this to be honest. Compiling on the Pi is painful to say the least, in particular since the eBPF module appears to still not work, which means a kernel recompile.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[fntlnz]

@vpetersson I'm closing since there were no dibs on this one and no progress in a long time.

Feel free to reopen if you think you can submit a PR to get ARM supported out of the box, we would love it!

The sysdig issue linked here (https://github.com/draios/sysdig/issues/1290) is still open since that is an actual issue that we have to solve and independent to the request of supporting another platform.

[krisnova]

Reopening - I am going to release a blog this week on how to do this. Please stay tuned.

[krisnova]

/kind feature

[krisnova]

See documentation in #1176 if you would like to do this now. Once it is merged I will do my best to continually test recent version of Falco to ensure we have basic arm support.

We can bring up creating arm packages on the next call.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.