oracle / bpftune

bpftune uses BPF to auto-tune Linux systems
Other
659 stars 56 forks source link

bpf(BPF_PROG_LOAD) -EINVAL #13

Closed pavlinux closed 1 year ago

pavlinux commented 1 year ago

.....

pavlinux commented 1 year ago

pftune -sD root@amd64:/media/kernel/bpftune/src# bpftune: bpftune works fully bpftune: bpftune supports per-netns policy (via netns cookie) bpftune: could not load skeleton: Invalid argument bpftune: error initializing '/usr/lib64/bpftune//route_table_tuner.so: Invalid argument bpftune: could not open /proc/sys/net/ipv6/neigh/default/gc_interval (netns fd 0) for reading: No such file or directory bpftune: error reading tunable 'net.ipv6.neigh.default.gc_interval': No such file or directory bpftune: error initializing '/usr/lib64/bpftune//neigh_table_tuner.so: No such file or directory bpftune: could not load skeleton: Invalid argument bpftune: error initializing '/usr/lib64/bpftune//tcp_cong_tuner.so: Invalid argument bpftune: could not load skeleton: Invalid argument bpftune: error initializing '/usr/lib64/bpftune//route_table_tuner.so: Invalid argument bpftune: could not open /proc/sys/net/ipv6/neigh/default/gc_interval (netns fd 0) for reading: No such file or directory bpftune: error reading tunable 'net.ipv6.neigh.default.gc_interval': No such file or directory bpftune: error initializing '/usr/lib64/bpftune//neigh_table_tuner.so: No such file or directory bpftune: could not load skeleton: Invalid argument bpftune: error initializing '/usr/lib64/bpftune//tcp_cong_tuner.so: Invalid argument

alan-maguire commented 1 year ago

thanks for the report. What distro is this (I see it's a 5.10 kernel from above)? Is kernel/module BTF available in /sys/kernel/btf ?

pavlinux commented 1 year ago

config.gz

bpftune.5.10.patch.gz

alan-maguire commented 1 year ago

do the above failures go away with ipv6 disabled as per other issue? if so, let's use this one to track better bpftune handling of the ipv6 disabled case..

pavlinux commented 1 year ago

do the above failures go away with ipv6 disabled as per other issue? if so, let's use this one to track better bpftune handling of the ipv6 disabled case..

  • Enabled IPv6 in the kernel
  • compiled as a module
  • net.ipv6.conf.default.disable_ipv6 = 0
    
    # bpftune -s

bpftune: bpftune works fully bpftune: bpftune supports per-netns policy (via netns cookie) bpftune: could not load skeleton: Invalid argument bpftune: error initializing '/usr/lib64/bpftune//tcp_cong_tuner.so: Invalid argument bpftune: could not load skeleton: No such process bpftune: error initializing '/usr/lib64/bpftune//route_table_tuner.so: No such process bpftune: could not load skeleton: Invalid argument bpftune: error initializing '/usr/local/lib64/bpftune//tcp_cong_tuner.so: Invalid argument bpftune: could not load skeleton: No such process bpftune: error initializing '/usr/local/lib64/bpftune//route_table_tuner.so: No such process

pavlinux commented 1 year ago
# bpftune -ds
bpftune: libbpf: prog 'entry__fib6_age': failed to find kernel BTF type ID of 'fib6_age': -3
bpftune: libbpf: prog 'entry__fib6_age': failed to prepare load attributes: -3
bpftune: libbpf: prog 'entry__fib6_age': failed to load: -3
bpftune: libbpf: failed to load object 'route_table_tuner_bpf'
bpftune: libbpf: failed to load BPF skeleton 'route_table_tuner_bpf': -3
bpftune: could not load skeleton: No such process
bpftune: drop caps (count 0)
bpftune: error initializing '/usr/local/lib64/bpftune//route_table_tuner.so: No such process
# bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep fib6_age
# echo $?
1
pavlinux commented 1 year ago

Write the requirements for the kernel config. Or handle all exceptions.

For example, the symbol fib6_age cannot be loaded from vmlinux because IPV6 is built as a module.

pavlinux commented 1 year ago
bpftune: searching /usr/lib64/bpftune/ for plugins...
bpftune: found lib /usr/lib64/bpftune//tcp_cong_tuner.so, init
bpftune: set caps (count 1)
bpftune: drop caps (count 0)
bpftune: calling init for '/usr/lib64/bpftune//tcp_cong_tuner.so
bpftune: set caps (count 1)
bpftune: could not init module '/lib/modules/5.10.186/kernel/net/ipv4/tcp_bbr.ko'
bpftune: drop caps (count 0)
bpftune: set caps (count 1)
# ls -la /lib/modules/5.10.186/kernel/net/ipv4/tcp_bbr.ko
-rw-r--r-- 1 root root 791440 Jul  7 19:38 /lib/modules/5.10.186/kernel/net/ipv4/tcp_bbr.ko
pavlinux commented 1 year ago
71: (07) r4 += -40
; if (bpf_getsockopt(ctx, SOL_TCP, TCP_CONGESTION, &buf, sizeof(buf)) ||
72: (bf) r1 = r6
73: (b7) r2 = 6
74: (b7) r3 = 13
75: (b7) r5 = 16
76: (85) call bpf_getsockopt#57
unknown func bpf_getsockopt#57
processed 68 insns (limit 1000000) max_states_per_insn 0 total_states 6 peak_states 6 mark_read 4
-- END PROG LOAD LOG --
bpftune: libbpf: prog 'bpftune_cong_iter': failed to load: -22
bpftune: libbpf: failed to load object 'tcp_cong_tuner_bpf'
bpftune: libbpf: failed to load BPF skeleton 'tcp_cong_tuner_bpf': -22
bpftune: could not load skeleton: Invalid argument
bpftune: drop caps (count 0)
bpftune: error initializing '/usr/lib64/bpftune//tcp_cong_tuner.so: Invalid argument
alan-maguire commented 1 year ago

bpf_getsockopt() is a BPF helper that's existed in the kernel since around the 4.19 kernel. i checked and bpf tcp iterator support was added prior to 5.10, but support for bpf_getsockopt was not added until 5.14 via commit 3cee6fb8e69e. i will adjust the support test to include this so bpftune falls back to legacy mode.

alan-maguire commented 1 year ago

above is done via https://github.com/oracle-samples/bpftune/pull/25

pavlinux commented 1 year ago

bpf_getsockopt() is a BPF helper that's existed in the kernel since around the 4.19 kernel. i checked and bpf tcp iterator support was added prior to 5.10, but support for bpf_getsockopt was not added until 5.14 via commit 3cee6fb8e69e. i will adjust the support test to include this so bpftune falls back to legacy mode.

Create patch for 5.10 from 5.15-rc1

bpf_get.set_sockopt_in_bpf_tcp_iter.5.10.patch.gz