[Modern eBPF] libpman: tracing program type is not supported

[kennylindberg]

When trying to start falco using modern eBPF, we get the following error:

# /usr/bin/falco --pidfile=/var/run/falco.pid --modern-bpf
[...]
Fri Sep  8 14:38:29 2023: One ring buffer every '2' CPUs.
libpman: tracing program type is not supported (errno: 22 | message: Invalid argument)
Fri Sep  8 14:38:29 2023: An error occurred in an event source, forcing termination...
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
Error:

Falco version: 0.35.1 OS: Oracle Linux 7.9 Kernel: 5.4.17-2136.322.6.2.el7uek.x86_64

Despite the kernel version is version 5.4, BPF ring buffer and BTF should be available:

# bpftool feature probe kernel | grep -q "map_type ringbuf is available" && echo "true" || echo "false"
true
# file /sys/kernel/btf/vmlinux
/sys/kernel/btf/vmlinux: data

Running a stracktrace seems like the kernel version isn't detected correctly (KERNEL_VERSION(0, 0, 0)):

bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, insn_cnt=2, insns=0x7fff51c782f0, license="GPL", log_level=1, log_size=4096, log_buf=" s\307Q\377\177", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=0x18 /* BPF_??? */, ...}, 128) = -1 EINVAL (Invalid argument)
write(2, "libpman: tracing program type is"..., 87libpman: tracing program type is not supported (errno: 22 | message: Invalid argument)

[Andreagit97]

It seems like BTF programs are not supported on your machine, try this:

sudo bpftool feature probe kernel | grep -q "program_type tracing is available" && echo "true" || echo "false" 

Unfortunately these https://falco.org/docs/event-sources/kernel/#requirements are essential requirements to run the modern bpf probe :/

[kennylindberg]

As far as I can see, BTF should be supported and compiled into the kernel:

# cat /boot/config-5.4.17-2136.322.6.2.el7uek.x86_64 | grep BTF
# CONFIG_VIDEO_SONY_BTF_MPX is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_PAHOLE_HAS_SPLIT_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y

The command you've sent is unreliable, as it returns false on my machine, but also returns false on our Oracle Linux 8 machines where the modern BPF actually works:

# bpftool feature probe kernel | grep "program_type tracing"
eBPF program_type tracing is NOT available
[root@localhost ~]# systemctl status falco-modern-bpf
● falco-modern-bpf.service - Falco: Container Native Runtime Security with modern ebpf
   Loaded: loaded (/usr/lib/systemd/system/falco-modern-bpf.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-09-11 08:46:08 CEST; 6min ago
     Docs: https://falco.org/docs/
 Main PID: 789 (falco)
    Tasks: 10 (limit: 22887)
   Memory: 55.5M
   CGroup: /system.slice/falco-modern-bpf.service
           └─789 /usr/bin/falco --pidfile=/var/run/falco.pid --modern-bpf

Sep 11 08:46:08 localhost.localdomain falco[789]: Falco version: 0.35.1 (x86_64)
Sep 11 08:46:08 localhost.localdomain falco[789]: Falco initialized with configuration file: /etc/falco/falco.yaml
Sep 11 08:46:08 localhost.localdomain falco[789]: Loading rules from file /etc/falco/falco_rules.yaml
Sep 11 08:46:08 localhost.localdomain falco[789]: Loading rules from file /etc/falco/falco_rules.local.yaml
Sep 11 08:46:08 localhost.localdomain falco[789]: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Sep 11 08:46:08 localhost.localdomain falco[789]: Starting health webserver with threadiness 4, listening on port 8765
Sep 11 08:46:08 localhost.localdomain falco[789]: Loaded event sources: syscall
Sep 11 08:46:08 localhost.localdomain falco[789]: Enabled event sources: syscall
Sep 11 08:46:08 localhost.localdomain falco[789]: Opening 'syscall' source with modern BPF probe.
Sep 11 08:46:08 localhost.localdomain falco[789]: One ring buffer every '2' CPUs.

[Andreagit97]

Having the BTF feature enabled on the system doesn't mean that all BTF programs are supported... let me explain better, there are several program types in BPF (https://docs.kernel.org/bpf/libbpf/program_types.html). In the modern probe, we need the prog type BPF_PROG_TYPE_TRACING but as you can notice the BPF_PROG_TYPE_TRACING can have different attach types:

+-------------------------------------------+----------------------------------------+----------------------------------+-----------+
| ``BPF_PROG_TYPE_TRACING``                 | ``BPF_MODIFY_RETURN``                  | ``fmod_ret+`` [#fentry]_         |           |
+                                           +                                        +----------------------------------+-----------+
|                                           |                                        | ``fmod_ret.s+`` [#fentry]_       | Yes       |
+                                           +----------------------------------------+----------------------------------+-----------+
|                                           | ``BPF_TRACE_FENTRY``                   | ``fentry+`` [#fentry]_           |           |
+                                           +                                        +----------------------------------+-----------+
|                                           |                                        | ``fentry.s+`` [#fentry]_         | Yes       |
+                                           +----------------------------------------+----------------------------------+-----------+
|                                           | ``BPF_TRACE_FEXIT``                    | ``fexit+`` [#fentry]_            |           |
+                                           +                                        +----------------------------------+-----------+
|                                           |                                        | ``fexit.s+`` [#fentry]_          | Yes       |
+                                           +----------------------------------------+----------------------------------+-----------+
|                                           | ``BPF_TRACE_ITER``                     | ``iter+`` [#iter]_               |           |
+                                           +                                        +----------------------------------+-----------+
|                                           |                                        | ``iter.s+`` [#iter]_             | Yes       |
+                                           +----------------------------------------+----------------------------------+-----------+
|                                           | ``BPF_TRACE_RAW_TP``                   | ``tp_btf+`` [#fentry]_           |           |
+-------------------------------------------+----------------------------------------+----------------------------------+-----------+

More in detail in the modern probe we need the BPF_TRACE_RAW_TP attach type but right now there are no available functions to probe this specific attach type, so what we do is an approximation, as you said. We probe the BPF_TRACE_FENTRY attach type since it is the only attach type we can probe in the kernel... So yes the check is not 100% reliable but we cannot do more than that :/

[kennylindberg]

Thank you for the detailed explanation.

Wouldn't it be possible just to probe this directly through the BTF sys fs export /sys/kernel/btf/vmlinux?

As you can see, the following tells me that the BPF type BPF_TRACE_RAW_TP is available:

# strings /sys/kernel/btf/vmlinux | grep -i BPF_TRACE_RAW_TP
BPF_TRACE_RAW_TP

Also, it tells me that the BPF type BPF_TRACE_FENTRY isn't available, why the modern BPF fails to start:

# strings /sys/kernel/btf/vmlinux | grep -i BPF_TRACE_FENTRY
(no output)

[Andreagit97]

Yeah, this seems a great idea! The best solution would probably be to implement in libbpf the possibility of probing our exact attach type BPF_TRACE_RAW_TP but at the moment the one you proposed could be a possible workaround! Unfortunately, we are really near the release so I don't think we can do that for Falco 0.36 :/

[kennylindberg]

That sounds great! We will look forward to that :)

[FedeDP]

Hi! I just released Falco 0.36.2-rc1 that should solve this issue; care to try it? You can use docker images or packages just like you would do for a normal Falco release!

[kennylindberg]

Sure! However, I don't see the RPM package to be available anywhere?

[FedeDP]

Oh yep because it got published under the rpm-dev repo: https://download.falco.org/packages/rpm-dev/falco-0.36.2-rc1-x86_64.rpm

[kennylindberg]

Ah thanks. It's still failing now with the following output:

Wed Oct 25 15:04:21 2023: One ring buffer every '2' CPUs.
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
; int BPF_PROG(sys_enter,
0: (bf) r6 = r1
; int BPF_PROG(sys_enter,
1: (79) r7 = *(u64 *)(r6 +8)
; return g_64bit_interesting_syscalls_table[syscall_id & (SYSCALL_TABLE_SIZE - 1)];
2: (bf) r1 = r7
3: (57) r1 &= 511
; return g_64bit_interesting_syscalls_table[syscall_id & (SYSCALL_TABLE_SIZE - 1)];
4: (18) r2 = 0xffffa92ec28ce000
6: (0f) r2 += r1
last_idx 6 first_idx 0
regs=2 stack=0 before 4: (18) r2 = 0xffffa92ec28ce000
regs=2 stack=0 before 3: (57) r1 &= 511
regs=2 stack=0 before 2: (bf) r1 = r7
regs=80 stack=0 before 1: (79) r7 = *(u64 *)(r6 +8)
7: (71) r2 = *(u8 *)(r2 +0)
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=map_value(id=0,off=0,ks=4,vs=242537,umax_value=511,var_off=(0x0; 0x1ff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; if(!syscalls_dispatcher__64bit_interesting_syscall(syscall_id))
8: (15) if r2 == 0x0 goto pc+76
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; return g_settings.dropping_mode;
9: (18) r2 = 0xffffa92ec28ce5a8
11: (71) r2 = *(u8 *)(r2 +12)
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=map_value(id=0,off=1448,ks=4,vs=242537,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; if(!maps__get_dropping_mode())
12: (15) if r2 == 0x0 goto pc+37
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; return g_64bit_sampling_syscall_table[syscall_id & (SYSCALL_TABLE_SIZE - 1)];
13: (18) r2 = 0xffffa92ec28ce200
15: (0f) r2 += r1
16: (71) r1 = *(u8 *)(r2 +0)
 R1_w=invP(id=0,umax_value=511,var_off=(0x0; 0x1ff)) R2_w=map_value(id=0,off=512,ks=4,vs=242537,umax_value=511,var_off=(0x0; 0x1ff)) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv(id=0) R10=fp0
; if(sampling_flag == UF_NEVER_DROP)
17: (15) if r1 == 0x2 goto pc+32
 R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2=map_value(id=0,off=512,ks=4,vs=242537,umax_value=511,var_off=(0x0; 0x1ff)) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0) R10=fp0
18: (15) if r1 == 0x4 goto pc+66
 R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R2=map_value(id=0,off=512,ks=4,vs=242537,umax_value=511,var_off=(0x0; 0x1ff)) R6=ctx(id=0,off=0,imm=0) R7=inv(id=0) R10=fp0
; if((bpf_ktime_get_boot_ns() % SECOND_TO_NS) >= (SECOND_TO_NS / maps__get_sampling_ratio()))
19: (85) call bpf_uuused15#125
unknown func bpf_uuused15#125
processed 17 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
-- END PROG LOAD LOG --
libbpf: prog 'sys_enter': failed to load: -22
libbpf: failed to load object 'bpf_probe'
libbpf: failed to load BPF skeleton 'bpf_probe': -22
libpman: failed to load BPF object (errno: 22 | message: Invalid argument)
Wed Oct 25 15:04:21 2023: An error occurred in an event source, forcing termination...
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
Error: Initialization issues during scap_init

[FedeDP]

This seems a different problem, right?

19: (85) call bpf_uuused15#125 unknown func bpf_uuused15#125

I think the issue is because on 5.4 BPF_FUNC_ktime_get_boot_ns did not exist; but that is what we are using in the modern bpf probe since it was made to work against kenrels >= 5.8: https://github.com/falcosecurity/libs/blob/master/driver/modern_bpf/helpers/interfaces/attached_programs.h#L59. Perhaps we might need to make the same workaround we do for old bpf probe (https://github.com/falcosecurity/libs/blob/master/driver/bpf/bpf_helpers.h#L24)? I am not sure this is the reason of the failure anyway, let me ping @Andreagit97!

[Andreagit97]

Uhm yes, the issue seems the one described by @FedeDP, BTW I've never seen this bpf_uuused15... To be honest, this issue is due to some missing backporting on kernel 5.4.17-2136.322.6.2.el7uek.x86_64 so I think that the best solution here is not to use the modern bpf at all. We don't know which helpers are backported or not... the suggestion here is to switch to a recent kernel version ASAP or to use one of the other 2 drivers (legacy_bpf,kernel module) if possible

[kennylindberg]

All right, I see. Thank you for your help and the explanation.