Closed programmerq closed 1 week ago
AFAIK eBPF modules are verified on load, so if the program can run will will be loaded, otherwise it will be rejected by the kernel. Base on that I'd change our logic from: Check the kernel > 5.8 and prevent loading the module to Load the module and if we fail to load return "kernel 5.8 is required" error.
backported to v16, v15, v14
What would you like Teleport to do?
Currently, Teleport's kernel version check is a very simple semver comparison. Since RedHat tends to backport many newer kernel features to an older version, there can be kernels that would otherwise work with Teleport's BPF features.
Rather than creating a cherry-picked list of kernel version exceptions to allow the BPF feature to load, a flag or configuration option that allows a user to disable the version check would allow them to use Teleport's enhanced session recording and networking features.
The flag/feature should be properly caveated so anyone using it would be aware that they should exercise due diligence to make sure the kernel they are using actually works with our BPF code.
https://github.com/gravitational/teleport/blob/v15.2.2/lib/bpf/common.go#L104-L113
What problem does this solve?
In particular, the RedHat 8 kernel has many backported BPF features. I ran a test build of the Teleport binary with the version check commented out, and my cursory findings are that enhanced session recording works with 4.18.0-513.24.1.el8_9.x86_64.
Teleport 15.2.2 (modified) on 4.18.0-513.24.1.el8_9.x86_64 - libbpf debug output
```log [root@localhost ~]# journalctl -u teleport _PID=36045 | grep -iC2 bpf Apr 23 19:16:31 localhost.localdomain teleport[36045]: {"caller":"cgroup/cgroup.go:323","component":"cgroups","level":"debug","message":"Mounted cgroup filesystem to /cgroup2.","timestamp":"2024-04-23T19:16:31Z"} Apr 23 19:16:31 localhost.localdomain teleport[36045]: {"caller":"cgroup/cgroup.go:100","component":"cgroups","level":"debug","message":"Teleport session hierarchy mounted at: /cgroup2/teleport/a790671c-1492-015d-114b-39f547c0af60.","timestamp":"2024-04-23T19:16:31Z"} Apr 23 19:16:31 localhost.localdomain teleport[36045]: {"caller":"bpf/bpf.go:166","component":"bpf","level":"debug","message":"Starting enhanced session recording.","timestamp":"2024-04-23T19:16:31Z"} Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: loading object 'command' from buffer Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(2) .text, size 2704, link 0, flags 6, type=1 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.text': found program 'enter_execve' at insn offset 0 (0 bytes), code size 267 insns (2136 bytes) Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.text': found program 'exit_execve' at insn offset 267 (2136 bytes), code size 71 insns (568 bytes) Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(3) .rel.text, size 192, link 19, flags 40, type=9 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(4) tp/syscalls/sys_execve, size 40, link 0, flags 6, type=1 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec 'tp/syscalls/sys_execve': found program 'tracepoint__syscalls__sys_enter_execve' at insn offset 0 (0 bytes), code size 5 insns (40 bytes) Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(5) .reltp/syscalls/sys_execve, size 16, link 19, flags 40, type=9 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(6) tp/syscalls/sys_exit_execve, size 32, link 0, flags 6, type=1 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec 'tp/syscalls/sys_exit_execve': found program 'tracepoint__syscalls__sys_exit_execve' at insn offset 0 (0 bytes), code size 4 insns (32 bytes) Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(7) .reltp/syscalls/sys_exit_execve, size 16, link 19, flags 40, type=9 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(8) tp/syscalls/sys_execveat, size 40, link 0, flags 6, type=1 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec 'tp/syscalls/sys_execveat': found program 'tracepoint__syscalls__sys_enter_execveat' at insn offset 0 (0 bytes), code size 5 insns (40 bytes) Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(9) .reltp/syscalls/sys_execveat, size 16, link 19, flags 40, type=9 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(10) tp/syscalls/sys_exit_execveat, size 32, link 0, flags 6, type=1 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec 'tp/syscalls/sys_exit_execveat': found program 'tracepoint__syscalls__sys_exit_execveat' at insn offset 0 (0 bytes), code size 4 insns (32 bytes) Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(11) .reltp/syscalls/sys_exit_execveat, size 16, link 19, flags 40, type=9 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(12) license, size 13, link 0, flags 3, type=1 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: license of command is Dual BSD/GPL Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(13) .maps, size 96, link 0, flags 3, type=1 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(14) .BTF, size 28567, link 0, flags 0, type=1 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(16) .BTF.ext, size 4004, link 0, flags 0, type=1 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: elf: section(19) .symtab, size 696, link 1, flags 0, type=2 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: looking for externs among 29 symbols... Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: collected 0 externs total Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'monitored_cgroups': at sec_idx 13, offset 0. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'monitored_cgroups': found type = 1. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'monitored_cgroups': found key [8], sz = 8. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'monitored_cgroups': found value [12], sz = 8. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'monitored_cgroups': found max_entries = 1024. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'execve_events': at sec_idx 13, offset 32. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'execve_events': found type = 27. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'execve_events': found max_entries = 32768. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_counter': at sec_idx 13, offset 48. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_counter': found type = 2. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_counter': found key [27], sz = 4. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_counter': found value [8], sz = 8. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_counter': found max_entries = 1. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_doorbell': at sec_idx 13, offset 80. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_doorbell': found type = 27. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_doorbell': found max_entries = 4096. Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': collecting relocation for section(2) '.text' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #0: insn #9 against 'monitored_cgroups' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'enter_execve': found map 0 (monitored_cgroups, sec 13, off 0) for insn #9 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #1: insn #42 against 'execve_events' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'enter_execve': found map 1 (execve_events, sec 13, off 32) for insn #42 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #2: insn #61 against 'lost_counter' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'enter_execve': found map 2 (lost_counter, sec 13, off 48) for insn #61 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #3: insn #117 against 'execve_events' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'enter_execve': found map 1 (execve_events, sec 13, off 32) for insn #117 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #4: insn #179 against 'execve_events' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'enter_execve': found map 1 (execve_events, sec 13, off 32) for insn #179 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #5: insn #199 against 'lost_counter' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'enter_execve': found map 2 (lost_counter, sec 13, off 48) for insn #199 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #6: insn #210 against 'lost_counter' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'enter_execve': found map 2 (lost_counter, sec 13, off 48) for insn #210 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #7: insn #219 against 'lost_doorbell' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'enter_execve': found map 3 (lost_doorbell, sec 13, off 80) for insn #219 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #8: insn #272 against 'monitored_cgroups' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'exit_execve': found map 0 (monitored_cgroups, sec 13, off 0) for insn #5 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #9: insn #277 against 'execve_events' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'exit_execve': found map 1 (execve_events, sec 13, off 32) for insn #10 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #10: insn #287 against 'lost_counter' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'exit_execve': found map 2 (lost_counter, sec 13, off 48) for insn #20 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.rel.text': relo #11: insn #297 against 'lost_doorbell' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'exit_execve': found map 3 (lost_doorbell, sec 13, off 80) for insn #30 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.reltp/syscalls/sys_execve': collecting relocation for section(4) 'tp/syscalls/sys_execve' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.reltp/syscalls/sys_execve': relo #0: insn #2 against '.text' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.reltp/syscalls/sys_exit_execve': collecting relocation for section(6) 'tp/syscalls/sys_exit_execve' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.reltp/syscalls/sys_exit_execve': relo #0: insn #1 against '.text' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.reltp/syscalls/sys_execveat': collecting relocation for section(8) 'tp/syscalls/sys_execveat' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.reltp/syscalls/sys_execveat': relo #0: insn #2 against '.text' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.reltp/syscalls/sys_exit_execveat': collecting relocation for section(10) 'tp/syscalls/sys_exit_execveat' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec '.reltp/syscalls/sys_exit_execveat': relo #0: insn #1 against '.text' Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'monitored_cgroups': created successfully, fd=13 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'execve_events': created successfully, fd=14 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_counter': created successfully, fd=15 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: map 'lost_doorbell': created successfully, fd=16 Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: sec 'tp/syscalls/sys_execve': found 2 CO-RE relocations Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: CO-RE relocating [37] struct trace_event_raw_sys_enter: found target candidate [5992] struct trace_event_raw_sys_enter in [vmlinux] Apr 23 19:16:31 localhost.localdomain teleport[36045]: libbpf: prog 'tracepoint__syscalls__sys_enter_execve': relo #0:Users who are using Rhel8 could benefit from enhanced session recording without having to upgrade their fleet to use a different kernel.
If a workaround exists, please include it.
There is not a good workaround since it requires a code change.