Open lyuts opened 10 months ago
What kernel are you running on? Newer kernels (>= 5.9) will unload xdp programs when the creating process terminates, older kernels won't.
I'm on 6.1
$ uname -a
Linux btfbpf 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
Odd, can you run your userspace program under strace -e bpf and paste the output?
user@btfbpf:~/ws/aya-xdp$ sudo !!
sudo strace -e bpf ./target/debug/aya-xdp
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\20\0\0\0\20\0\0\0\5\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=45, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\r\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=93, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\r\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=93, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\17\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=95, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\f\0\0\0\f\0\0\0\7\0\0\0\1\0\0\0\0\0\0\20"..., btf_log_buf=NULL, btf_size=43, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0000\0\0\0000\0\0\0\22\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=90, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0(\0\0\0(\0\0\0\5\0\0\0\0\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=69, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\30\0\0\0\30\0\0\0\n\0\0\0\1\0\0\0\1\0\0\223"..., btf_log_buf=NULL, btf_size=58, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x559e5fbdbff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="aya_name_check", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=6, insns=0x559e5fbcd7f0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=2, insns=0x559e5fbcbf50, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 9
bpf(BPF_LINK_CREATE, {link_create={prog_fd=9, target_fd=-1, attach_type=BPF_PERF_EVENT, flags=0, perf_event={bpf_cookie=0}}}, 144) = -1 EBADF (Bad file descriptor)
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=32, max_entries=1, map_flags=0, inner_map_fd=0, map_name="aya_global", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=5, insns=0x559e5fbd2ae0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 10
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=2, insns=0x559e5fbcbf50, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_CPUMAP, key_size=4, value_size=8, max_entries=1, map_flags=0, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_DEVMAP, key_size=4, value_size=8, max_entries=1, map_flags=0, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=8192, max_entries=1, map_flags=0, inner_map_fd=0, map_name="AYA_LOG_BUF", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=4, map_flags=0, inner_map_fd=0, map_name="AYA_LOGS", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 10
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1024, map_flags=0, inner_map_fd=0, map_name="BLOCKLIST", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 11
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=55, max_entries=1, map_flags=BPF_F_RDONLY_PROG, inner_map_fd=0, map_name=".rodata", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 12
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=12, key=0x559e5e5f90dc, value=0x559e5fbcb2c0, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_FREEZE, {map_fd=12}, 144) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=10, key=0x7ffc75c404ec, value=0x7ffc75c405c4, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=10, key=0x7ffc75c404ec, value=0x7ffc75c405c4, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=10, key=0x7ffc75c404ec, value=0x7ffc75c405c4, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=10, key=0x7ffc75c404ec, value=0x7ffc75c405c4, flags=BPF_ANY}, 144) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_XDP, insn_cnt=911, insns=0x559e5fc231f0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(6, 1, 55), prog_flags=0, prog_name="aya_xdp", prog_ifindex=0, expected_attach_type=BPF_XDP, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 17
bpf(BPF_LINK_CREATE, {link_create={prog_fd=17, target_fd=2, attach_type=BPF_XDP, flags=0}}, 144) = 18
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=11, key=0x7ffc75c410b0, value=0x7ffc75c410b4, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=11, key=0x7ffc75c410b0, value=0x7ffc75c410b4, flags=BPF_ANY}, 144) = 0
^Cstrace: Process 7383 detached
It is using BPF_LINK_CREATE
as expected, so it should unlink on process termination. Or that was the case in 5.x at least, I'll have to check if behavior was changed. We have lifecycle integration tests here https://github.com/aya-rs/aya/blob/main/test/integration-test/src/tests/load.rs but we don't test killing a process, we probably should
Thank you for the pointers. Let me get familiar with aya tests as well as kernel behavior.
Quick follow up question, re: BPF_LINK_CREATE, is this the step after which a file is created in /sys/fs/bpf/
? I have the program running, but that directory stays always empty:
user@btfbpf:~/ws/aya-xdp$ sudo find /sys/fs/bpf -type f
user@btfbpf:~/ws/aya-xdp$
(iirc correctly from other articles, guides, some files should exists there during the program execution.
Update: I think I'm mixing this with 'pinning'. If so, please ignore this.
Correct, it’s pinning that creates files in bpffs
Tried running integration tests, with fresh main
got 3 test cases fail. Test cases are for relocation, so this may be a different issue.
failures:
---- tests::btf_relocations::relocation_tests::_enum_signed_32_true_some_kernelversion_new_6_0_0_https_git-
thread 'tests::btf_relocations::relocation_tests::_enum_signed_32_true_some_kernelversion_new_6_0_0_https_:
assertion `left == right` failed
left: 2219066437
right: 18446744071633650757
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- tests::btf_relocations::relocation_tests::_enum_signed_64_true_some_kernelversion_new_6_0_0_https_git-
thread 'tests::btf_relocations::relocation_tests::_enum_signed_64_true_some_kernelversion_new_6_0_0_https_:
assertion `left == right` failed
left: 572662307
right: 17524406869737742883
---- tests::btf_relocations::relocation_tests::_enum_unsigned_64_true_some_kernelversion_new_6_0_0_https_g-
thread 'tests::btf_relocations::relocation_tests::_enum_unsigned_64_true_some_kernelversion_new_6_0_0_http:
assertion `left == right` failed
left: 3722304989
right: 14757395259253972445
failures:
tests::btf_relocations::relocation_tests::_enum_signed_32_true_some_kernelversion_new_6_0_0_https_giths
tests::btf_relocations::relocation_tests::_enum_signed_64_true_some_kernelversion_new_6_0_0_https_giths
tests::btf_relocations::relocation_tests::_enum_unsigned_64_true_some_kernelversion_new_6_0_0_https_gis
what kernel are you running on and what command did you run to run the tests?
I'm on 6.1, and ran tests as:
cargo xtask integration-test vm /boot/vmlinuz-6.1.0-12-amd64
Hi Team,
Thank you for a very handy library. I'm new to aya and eBPF in general. I'm trying to figure out whether my observations are expected, or I am missing some chunk of code that would properly cleanup after program termination. I created a project from the aya template and implemented an xdp program following the guide in the aya book. The program was executed several times, and it either got terminated by me, or panic'd. When I run
bpftool prog
I keep seeing entries for all of the times the program was run.Is this expected? My impression is that this command would be showing only currently active programs. If the above is a sign of not properly cleaning up program's resources, could you point me to how cleanup is generally done with aya?
Any bit of information is appreciated. Thank you.