Tracee has too many errors when running in v6.5 with a real world cmdline/policy

[rafaeldtinoco]

Description

From https://github.com/aquasecurity/tracee/issues/3357#issuecomment-1838621610

$ sudo ./dist/tracee --no-containers -p ~/work/cli/test/ -o option:exec-env -o option:e
xec-hash -o option:parse-arguments -o option:sort-events -o none
{"level":"warn","ts":1701694975.813535,"msg":"libbpf: prog 'trace_ret_layout_and_allocate': failed to create kretprobe 'layout_and_allocate+0x0' perf event: No such file or directory"}
{"level":"error","ts":1701694975.8135998,"msg":"Event canceled because of missing probe dependency","missing probe":102,"event":"hidden_kernel_module_seeker"}
{"level":"warn","ts":1701694975.9879267,"msg":"libbpf: prog 'trace_ret_exec_binprm': failed to create kretprobe 'exec_binprm+0x0' perf event: No such file or directory"}
{"level":"error","ts":1701694975.9879916,"msg":"Event canceled because of missing probe dependency","missing probe":96,"event":"process_execute_failed"}
{"level":"warn","ts":1701694976.1228976,"msg":"libbpf: prog 'trace_load_elf_phdrs': failed to create kprobe 'load_elf_phdrs+0x0' perf event: Cannot assign requested address"}
{"level":"warn","ts":1701694976.2762296,"msg":"libbpf: prog 'trace_exec_binprm': failed to create kprobe 'exec_binprm+0x0' perf event: No such file or directory"}
{"level":"error","ts":1701694976.27629,"msg":"Event canceled because of missing probe dependency","missing probe":95,"event":"process_execute_failed"}
{"level":"warn","ts":1701694979.5936491,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 0: no address or symbols were provided to print_mem_dump event. please provide it via -e print_mem_dump.args.address=<hex address>, -e print_mem_dump.args.symbol_name=<owner>:<symbol> or -e print_mem_dump.args.symbol_name=<symbol> if specifying a system owned symbol"}
{"level":"warn","ts":1701694979.5936863,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 1: invalid symbols provided to print_mem_dump event: {system_,sys_,__x64_sys_,__arm64_sys_}compat_filldir64"}
{"level":"warn","ts":1701694979.5936952,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 1: invalid symbols provided to print_mem_dump event: {system_,sys_,__x64_sys_,__arm64_sys_} "}
{"level":"warn","ts":1701694979.5937037,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 2: no address or symbols were provided to print_mem_dump event. please provide it via -e print_mem_dump.args.address=<hex address>, -e print_mem_dump.args.symbol_name=<owner>:<symbol> or -e print_mem_dump.args.symbol_name=<symbol> if specifying a system owned symbol"}
{"level":"warn","ts":1701694979.5937123,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 3: no address or symbols were provided to print_mem_dump event. please provide it via -e print_mem_dump.args.address=<hex address>, -e print_mem_dump.args.symbol_name=<owner>:<symbol> or -e print_mem_dump.args.symbol_name=<symbol> if specifying a system owned symbol"}

In here we are seeing multiple issues when running Tracee in a v6.5 kernel:

1. symbols from the "hidden kernel module" logic that are required and don't exit https://github.com/aquasecurity/tracee/issues/3356
2. cancelled event "error" (should it be an error ?)
3. process_execute fails due to the lack of "exec_binprm" hook. https://github.com/aquasecurity/tracee/issues/3356
4. load_elf_phdrs fails due to the lack of missing hook. https://github.com/aquasecurity/tracee/issues/3653
5. bad symbols being used/given to print_mem_dump

[rafaeldtinoco]

This should be addressed together with https://github.com/aquasecurity/tracee/issues/3357 (when tracee starts testing all non syscall events on its e2e testing).

[geyslan]

On my env:

Linux hb 6.5.13-1-MANJARO #1 SMP PREEMPT_DYNAMIC Tue Nov 28 20:33:05 UTC 2023 x86_64 GNU/Linux clang version 14.0.6

sudo ./dist/tracee --no-containers -p ../policies/ -o option:exec-env -o option:exec-hash -o option:parse-arguments -o option:sort-events -o none
{"level":"warn","ts":1701697843.4055243,"msg":"libbpf: prog 'trace_exec_binprm': failed to create kprobe 'exec_binprm+0x0' perf event: No such file or directory"}
{"level":"error","ts":1701697843.4055736,"msg":"Event canceled because of missing probe dependency","missing probe":95,"event":"process_execute_failed"}
{"level":"warn","ts":1701697843.6334422,"msg":"libbpf: prog 'trace_load_elf_phdrs': failed to create kprobe 'load_elf_phdrs+0x0' perf event: Cannot assign requested address"}
{"level":"warn","ts":1701697843.6407342,"msg":"libbpf: prog 'trace_ret_layout_and_allocate': failed to create kretprobe 'layout_and_allocate+0x0' perf event: No such file or directory"}
{"level":"error","ts":1701697843.640771,"msg":"Event canceled because of missing probe dependency","missing probe":102,"event":"hidden_kernel_module_seeker"}
{"level":"warn","ts":1701697843.6488373,"msg":"libbpf: prog 'trace_ret_exec_binprm': failed to create kretprobe 'exec_binprm+0x0' perf event: No such file or directory"}
{"level":"error","ts":1701697843.6488798,"msg":"Event canceled because of missing probe dependency","missing probe":96,"event":"process_execute_failed"}
{"level":"warn","ts":1701697846.4451187,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 3: no address or symbols were provided to print_mem_dump event. please provide it via -e print_mem_dump.args.address=<hex address>, -e print_mem_dump.args.symbol_name=<owner>:<symbol> or -e print_mem_dump.args.symbol_name=<symbol> if specifying a system owned symbol"}
{"level":"warn","ts":1701697846.445153,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 0: no address or symbols were provided to print_mem_dump event. please provide it via -e print_mem_dump.args.address=<hex address>, -e print_mem_dump.args.symbol_name=<owner>:<symbol> or -e print_mem_dump.args.symbol_name=<symbol> if specifying a system owned symbol"}
{"level":"warn","ts":1701697846.4451606,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 1: invalid symbols provided to print_mem_dump event: {system_,sys_,__x64_sys_,__arm64_sys_} "}
{"level":"warn","ts":1701697846.4451668,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 1: invalid symbols provided to print_mem_dump event: {system_,sys_,__x64_sys_,__arm64_sys_}compat_filldir64"}
{"level":"warn","ts":1701697846.4451733,"msg":"Memory dump","error":"ebpf.(*Tracee).triggerMemDump: policy 2: no address or symbols were provided to print_mem_dump event. please provide it via -e print_mem_dump.args.address=<hex address>, -e print_mem_dump.args.symbol_name=<owner>:<symbol> or -e print_mem_dump.args.symbol_name=<symbol> if specifying a system owned symbol"}
{"level":"error","ts":1701697848.4511175,"msg":"Unspecifed BPF log","id":0,"type":"BPF_LOG_ID_UNSPEC","ret":4,"cpu":2,"file":"./pkg/ebpf/c/tracee.bpf.c","line":1091,"count":1}

There's a bpf log error being emitted: "file":"./pkg/ebpf/c/tracee.bpf.c","line":1091

[geyslan]

On Linux ubuntu-jammy 5.15.0-89-generic we have only the warns about print_mem_dump.

[geyslan]

So far I've detected on the newer kernel that walk_mod_tree() never reaches

I ponder that's related to the iteration limit (600).

I'll continue later this debugging focusing on the "missing probe" errors.

[geyslan]

We already have one issue related to the missing symbols (not inlined): https://github.com/aquasecurity/tracee/issues/3356