oracle / bpftune

bpftune uses BPF to auto-tune Linux systems
Other
654 stars 55 forks source link

bpftune on 6.5.0-1022-gcp #90

Open andrey-admin opened 3 weeks ago

andrey-admin commented 3 weeks ago

Hello,

Trying run bpftune on google instance on 6.5.0-1022-gcp kernel. Got this on logs:

Jun 14 11:58:59 nginx-08 bpftune[38936]: bpftune works, but no BPF Type Format information (BTF) is available. This means kernel data structure of fsets may not match those at compile-time, and tuners may not operate as expected. This mode of operation is unsupported, and failures are expected , so be warned. Jun 14 11:58:59 nginx-08 bpftune[38936]: bpftune supports per-netns policy (via netns cookie) Jun 14 11:58:59 nginx-08 bpftune[38936]: could not load skeleton: Kernel verifier blocks program loading Jun 14 11:58:59 nginx-08 bpftune[38936]: error initializing '/usr/lib64/bpftune//tcp_conn_tuner.so: Unknown error 4007 Jun 14 11:58:59 nginx-08 bpftune[38936]: could not open /proc/sys/net/ipv6/neigh/default/gc_interval (netns fd 0) for reading: No such file or directory Jun 14 11:58:59 nginx-08 bpftune[38936]: could not open /proc/sys/net/ipv6/neigh/default/gc_stale_time (netns fd 0) for reading: No such file or directory Jun 14 11:58:59 nginx-08 bpftune[38936]: could not open /proc/sys/net/ipv6/neigh/default/gc_thresh1 (netns fd 0) for reading: No such file or directory Jun 14 11:58:59 nginx-08 bpftune[38936]: could not open /proc/sys/net/ipv6/neigh/default/gc_thresh2 (netns fd 0) for reading: No such file or directory Jun 14 11:58:59 nginx-08 bpftune[38936]: could not open /proc/sys/net/ipv6/neigh/default/gc_thresh3 (netns fd 0) for reading: No such file or directory Jun 14 11:59:00 nginx-08 bpftune[38936]: could not open /proc/sys/net/ipv6/route/max_size (netns fd 0) for reading: No such file or directory Jun 14 11:59:00 nginx-08 bpftune[38936]: could not open /proc/sys/net/ipv6/ip6frag_high_thresh (netns fd 0) for reading: No such file or directory Jun 14 11:59:00 nginx-08 bpftune[38936]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput Jun 14 11:59:00 nginx-08 bpftune[38936]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (16777217 0 300) -> (16777217 0 375) Jun 14 11:59:00 nginx-08 bpftune[38936]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput Jun 14 11:59:00 nginx-08 bpftune[38936]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (16777217 0 300) -> (16777217 0 375) Jun 14 11:59:00 nginx-08 bpftune[38936]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput Jun 14 11:59:00 nginx-08 bpftune[38936]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (16777217 0 300) -> (16777217 0 375) Jun 14 11:59:00 nginx-08 bpftune[38936]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput Jun 14 11:59:00 nginx-08 bpftune[38936]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (16777217 0 300) -> (16777217 0 375)

How that can be fixed?

Thanks!

alan-maguire commented 3 weeks ago

the first thing to note above is

bpftune[38936]: bpftune works, but no BPF Type Format information (BTF) is available. This means kernel data structure of fsets may not match those at compile-time, and tuners may not operate as expected. This mode of operation is unsupported, and failures are expected , so be warned.

Can you check if /sys/kernel/btf/vmlinux is present? If not this makes bpftune operation difficult since it's relying on compiled-in offsets. In the absence of BTF we can get weird results like the tcp buffer rmem values above.

andrey-admin commented 3 weeks ago

ls -lh /sys/kernel/btf/vmlinux

-r--r--r-- 1 root root 5.7M Jun 14 11:56 /sys/kernel/btf/vmlinux

alan-maguire commented 3 weeks ago

okay can you run "bpftune -dsS" to get full debug logging? we appear to be falling back to "no BTF" support level and I'd like to understand why. One possibility is BTF can't be read - in some cases the libbpf version used to build bpftune is too old so cannot parse the available BTF.

andrey-admin commented 3 weeks ago

bpftune: set caps (count 1) bpftune: set caps (count 2) bpftune: drop caps (count 1) bpftune: set caps (count 2) bpftune: libbpf: loading object 'probe_bpf' from buffer bpftune: libbpf: elf: section(3) fentry/setup_net, size 88, link 0, flags 6, type=1 bpftune: libbpf: sec 'fentry/setup_net': found program 'entrysetup_net' at insn offset 0 (0 bytes), code size 11 insns (88 bytes) bpftune: libbpf: elf: section(4) iter/tcp, size 128, link 0, flags 6, type=1 bpftune: libbpf: sec 'iter/tcp': found program 'probe_cong_iter' at insn offset 0 (0 bytes), code size 16 insns (128 bytes) bpftune: libbpf: elf: section(5) tp_btf/neigh_create, size 16, link 0, flags 6, type=1 bpftune: libbpf: sec 'tp_btf/neigh_create': found program 'bpftune_neigh_create' at insn offset 0 (0 bytes), code size 2 insns (16 bytes) bpftune: libbpf: elf: section(6) cgroup/sysctl, size 16, link 0, flags 6, type=1 bpftune: libbpf: sec 'cgroup/sysctl': found program 'sysctl_write' at insn offset 0 (0 bytes), code size 2 insns (16 bytes) bpftune: libbpf: elf: section(7) license, size 7, link 0, flags 3, type=1 bpftune: libbpf: license of probe_bpf is GPL v2 bpftune: libbpf: elf: section(8) .rodata, size 24, link 0, flags 2, type=1 bpftune: libbpf: elf: section(9) .bss, size 25, link 0, flags 3, type=8 bpftune: libbpf: elf: section(10) .maps, size 128, link 0, flags 3, type=1 bpftune: libbpf: elf: section(21) .BTF, size 43773, link 0, flags 0, type=1 bpftune: libbpf: elf: section(23) .BTF.ext, size 444, link 0, flags 0, type=1 bpftune: libbpf: elf: section(31) .symtab, size 792, link 1, flags 0, type=2 bpftune: libbpf: looking for externs among 33 symbols... bpftune: libbpf: collected 0 externs total bpftune: libbpf: map 'ring_buffer_map': at sec_idx 10, offset 0. bpftune: libbpf: map 'ring_buffer_map': found type = 27. bpftune: libbpf: map 'ring_buffer_map': found max_entries = 131072. bpftune: libbpf: map 'netns_map': at sec_idx 10, offset 16. bpftune: libbpf: map 'netns_map': found type = 1. bpftune: libbpf: map 'netns_map': found key [12], sz = 8. bpftune: libbpf: map 'netns_map': found value [12], sz = 8. bpftune: libbpf: map 'netns_map': found max_entries = 65536. bpftune: libbpf: map 'netns_map': found map_flags = 0. bpftune: libbpf: map 'last_event_map': at sec_idx 10, offset 56. bpftune: libbpf: map 'last_event_map': found type = 1. bpftune: libbpf: map 'last_event_map': found key [12], sz = 8. bpftune: libbpf: map 'last_event_map': found value [12], sz = 8. bpftune: libbpf: map 'last_event_map': found max_entries = 65536. bpftune: libbpf: map 'probe_hash_map': at sec_idx 10, offset 88. bpftune: libbpf: map 'probe_hash_map': found type = 1. bpftune: libbpf: map 'probe_hash_map': found key [12], sz = 8. bpftune: libbpf: map 'probe_hash_map': found value [12], sz = 8. bpftune: libbpf: map 'probe_hash_map': found max_entries = 65536. bpftune: libbpf: map 'probe_hash_map': found map_flags = 0. bpftune: libbpf: map 'probe_bp.rodata' (global data): at sec_idx 8, offset 0, flags 480. bpftune: libbpf: map 4 is "probe_bp.rodata" bpftune: libbpf: map 'probe_bp.bss' (global data): at sec_idx 9, offset 0, flags 400. bpftune: libbpf: map 5 is "probe_bp.bss" bpftune: libbpf: Unsupported BTF_KIND:19 bpftune: libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': -22 bpftune: libbpf: failed to find valid kernel BTF bpftune: libbpf: Error loading vmlinux BTF: -3 bpftune: libbpf: failed to load object 'probe_bpf' bpftune: libbpf: failed to load BPF skeleton 'probe_bpf': -3 bpftune: full bpftune support not available: Unknown error -3 bpftune: libbpf: loading object 'probe_bpf_legacy' from buffer bpftune: libbpf: elf: section(3) kprobe/setup_net, size 88, link 0, flags 6, type=1 bpftune: libbpf: sec 'kprobe/setup_net': found program 'entrysetup_net' at insn offset 0 (0 bytes), code size 11 insns (88 bytes) bpftune: libbpf: elf: section(4) raw_tracepoint/neigh_create, size 16, link 0, flags 6, type=1 bpftune: libbpf: sec 'raw_tracepoint/neigh_create': found program 'bpftune_neigh_create' at insn offset 0 (0 bytes), code size 2 insns (16 bytes) bpftune: libbpf: elf: section(5) cgroup/sysctl, size 16, link 0, flags 6, type=1 bpftune: libbpf: sec 'cgroup/sysctl': found program 'sysctl_write' at insn offset 0 (0 bytes), code size 2 insns (16 bytes) bpftune: libbpf: elf: section(6) license, size 7, link 0, flags 3, type=1 bpftune: libbpf: license of probe_bpf_legacy is GPL v2 bpftune: libbpf: elf: section(7) .rodata, size 24, link 0, flags 2, type=1 bpftune: libbpf: elf: section(8) .bss, size 25, link 0, flags 3, type=8 bpftune: libbpf: elf: section(9) .maps, size 128, link 0, flags 3, type=1 bpftune: libbpf: elf: section(20) .BTF, size 43608, link 0, flags 0, type=1 bpftune: libbpf: elf: section(22) .BTF.ext, size 284, link 0, flags 0, type=1 bpftune: libbpf: elf: section(30) .symtab, size 720, link 1, flags 0, type=2 bpftune: libbpf: looking for externs among 30 symbols... bpftune: libbpf: collected 0 externs total bpftune: libbpf: map 'ring_buffer_map': at sec_idx 9, offset 0. bpftune: libbpf: map 'ring_buffer_map': found type = 27. bpftune: libbpf: map 'ring_buffer_map': found max_entries = 131072. bpftune: libbpf: map 'netns_map': at sec_idx 9, offset 16. bpftune: libbpf: map 'netns_map': found type = 1. bpftune: libbpf: map 'netns_map': found key [12], sz = 8. bpftune: libbpf: map 'netns_map': found value [12], sz = 8. bpftune: libbpf: map 'netns_map': found max_entries = 65536. bpftune: libbpf: map 'netns_map': found map_flags = 0. bpftune: libbpf: map 'last_event_map': at sec_idx 9, offset 56. bpftune: libbpf: map 'last_event_map': found type = 1. bpftune: libbpf: map 'last_event_map': found key [12], sz = 8. bpftune: libbpf: map 'last_event_map': found value [12], sz = 8. bpftune: libbpf: map 'last_event_map': found max_entries = 65536. bpftune: libbpf: map 'probe_hash_map': at sec_idx 9, offset 88. bpftune: libbpf: map 'probe_hash_map': found type = 1. bpftune: libbpf: map 'probe_hash_map': found key [12], sz = 8. bpftune: libbpf: map 'probe_hash_map': found value [12], sz = 8. bpftune: libbpf: map 'probe_hash_map': found max_entries = 65536. bpftune: libbpf: map 'probe_hash_map': found map_flags = 0. bpftune: libbpf: map 'probe_bp.rodata' (global data): at sec_idx 7, offset 0, flags 480. bpftune: libbpf: map 4 is "probe_bp.rodata" bpftune: libbpf: map 'probe_bp.bss' (global data): at sec_idx 8, offset 0, flags 400. bpftune: libbpf: map 5 is "probe_bp.bss" bpftune: libbpf: Unsupported BTF_KIND:19 bpftune: libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': -22 bpftune: libbpf: failed to find valid kernel BTF bpftune: libbpf: Error loading vmlinux BTF: -3 bpftune: libbpf: failed to load object 'probe_bpf_legacy' bpftune: libbpf: failed to load BPF skeleton 'probe_bpf_legacy': -3 bpftune: legacy bpftune support not available: Unknown error -3 bpftune: libbpf: loading object 'probe_bpf_nobtf' from buffer bpftune: libbpf: elf: section(3) kprobe/setup_net, size 16, link 0, flags 6, type=1 bpftune: libbpf: sec 'kprobe/setup_net': found program 'entry__setup_net' at insn offset 0 (0 bytes), code size 2 insns (16 bytes) bpftune: libbpf: elf: section(4) raw_tracepoint/neigh_create, size 16, link 0, flags 6, type=1 bpftune: libbpf: sec 'raw_tracepoint/neigh_create': found program 'bpftune_neigh_create' at insn offset 0 (0 bytes), code size 2 insns (16 bytes) bpftune: libbpf: elf: section(5) cgroup/sysctl, size 16, link 0, flags 6, type=1 bpftune: libbpf: sec 'cgroup/sysctl': found program 'sysctl_write' at insn offset 0 (0 bytes), code size 2 insns (16 bytes) bpftune: libbpf: elf: section(6) license, size 7, link 0, flags 3, type=1 bpftune: libbpf: license of probe_bpf_nobtf is GPL v2 bpftune: libbpf: elf: section(7) .rodata, size 24, link 0, flags 2, type=1 bpftune: libbpf: elf: section(8) .bss, size 25, link 0, flags 3, type=8 bpftune: libbpf: elf: section(9) .maps, size 128, link 0, flags 3, type=1 bpftune: libbpf: elf: section(19) .BTF, size 2258, link 0, flags 0, type=1 bpftune: libbpf: elf: section(21) .BTF.ext, size 160, link 0, flags 0, type=1 bpftune: libbpf: elf: section(29) .symtab, size 672, link 1, flags 0, type=2 bpftune: libbpf: looking for externs among 28 symbols... bpftune: libbpf: collected 0 externs total bpftune: libbpf: map 'ring_buffer_map': at sec_idx 9, offset 0. bpftune: libbpf: map 'ring_buffer_map': found type = 27. bpftune: libbpf: map 'ring_buffer_map': found max_entries = 131072. bpftune: libbpf: map 'netns_map': at sec_idx 9, offset 16. bpftune: libbpf: map 'netns_map': found type = 1. bpftune: libbpf: map 'netns_map': found key [12], sz = 8. bpftune: libbpf: map 'netns_map': found value [12], sz = 8. bpftune: libbpf: map 'netns_map': found max_entries = 65536. bpftune: libbpf: map 'netns_map': found map_flags = 0. bpftune: libbpf: map 'last_event_map': at sec_idx 9, offset 56. bpftune: libbpf: map 'last_event_map': found type = 1. bpftune: libbpf: map 'last_event_map': found key [12], sz = 8. bpftune: libbpf: map 'last_event_map': found value [12], sz = 8. bpftune: libbpf: map 'last_event_map': found max_entries = 65536. bpftune: libbpf: map 'probe_hash_map': at sec_idx 9, offset 88. bpftune: libbpf: map 'probe_hash_map': found type = 1. bpftune: libbpf: map 'probe_hash_map': found key [12], sz = 8. bpftune: libbpf: map 'probe_hash_map': found value [12], sz = 8. bpftune: libbpf: map 'probe_hash_map': found max_entries = 65536. bpftune: libbpf: map 'probe_hash_map': found map_flags = 0. bpftune: libbpf: map 'probe_bp.rodata' (global data): at sec_idx 7, offset 0, flags 480. bpftune: libbpf: map 4 is "probe_bp.rodata" bpftune: libbpf: map 'probe_bp.bss' (global data): at sec_idx 8, offset 0, flags 400. bpftune: libbpf: map 5 is "probe_bp.bss" bpftune: libbpf: map 'ring_buffer_map': created successfully, fd=5 bpftune: libbpf: map 'netns_map': created successfully, fd=6 bpftune: libbpf: map 'last_event_map': created successfully, fd=7 bpftune: libbpf: map 'probe_hash_map': created successfully, fd=8 bpftune: libbpf: map 'probe_bp.rodata': created successfully, fd=9 bpftune: libbpf: map 'probe_bp.bss': created successfully, fd=10 bpftune: drop caps (count 1) bpftune: bpftune works, but no BPF Type Format information (BTF) is available. This means kernel data structure offsets may not match those at compile-time, and tuners may not operate as expected. This mode of operation is unsupported, and failures are expected, so be warned. bpftune: bpftune supports per-netns policy (via netns cookie)

andrey-admin commented 3 weeks ago

uname -a: Linux nginx-08 6.5.0-1022-gcp #24~22.04.1-Ubuntu SMP Tue May 28 16:34:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

alan-maguire commented 3 weeks ago

yep, as I suspected problem is with your BTF; the BTF generated by the kernel has BTF_KIND_ENUM64:

bpftune: libbpf: Unsupported BTF_KIND:19

....but the libbpf you used to build bpftune does not. As a result BTF in /sys/kernel/btf/vmlinux cannot be parsed. I'd suggest upgrading libbpf to > 1.0 since the pahole used to generate BTF must have used that version or newer.

So this isn't really an issue with bpftune itself; rather it's an issue with a mismatch between tools/libraries used to build the kernel and do BTF generation and the libraries we used to build bpftune.