undefined reference after last pull on ubuntu 22.04.2

[andrey-admin]

Trying rebuild after pull:

warning: Path diagnostic report is not generated. Current output format does not support diagnostics that cross file boundaries. Refer to --analyzer-output for valid output formats warning: Path diagnostic report is not generated. Current output format does not support diagnostics that cross file boundaries. Refer to --analyzer-output for valid output formats In file included from net_buffer_tuner.c:6: ./net_buffer_tuner.skel.h:197:2: warning: Potential leak of memory pointed to by 's' [unix.Malloc] return err; ^~~~~~ In file included from net_buffer_tuner.c:7: ./net_buffer_tuner.skel.legacy.h:197:2: warning: Potential leak of memory pointed to by 's' [unix.Malloc] return err; ^~~~~~ 2 warnings generated. cc -fPIC -Wall -Wextra -march=native -g -I../include -std=c99 -DBPFTUNE_VERSION='"5.19.0-1027-gcp"' -I../include -I/usr/include -I/usr/include/libnl3 -I../include/uapi -c libbpftune.c -o libbpftune.o cc -fPIC -Wall -Wextra -march=native -g -I../include -std=c99 -DBPFTUNE_VERSION='"5.19.0-1027-gcp"' -I../include -I/usr/include -I/usr/include/libnl3 -I../include/uapi -Wl,--version-script=libbpftune.map \ -Wl,--soname,libbpftune.so.0.1.3 \ -shared -o libbpftune.so.0.1.3 \ libbpftune.o \ -lbpf -ldl -lm -lrt -lcap -lnl-3 -lpthread -lnl-route-3 -L. -L/usr/local/lib64 ; \ rm -f libbpftune.so ; \ ln -sr libbpftune.so.0.1.3 libbpftune.so cc -fPIC -Wall -Wextra -march=native -g -I../include -std=c99 -DBPFTUNE_VERSION='"5.19.0-1027-gcp"' -I../include -I/usr/include -I/usr/include/libnl3 -I../include/uapi -c bpftune.c -o bpftune.o cc -fPIC -Wall -Wextra -march=native -g -I../include -std=c99 -DBPFTUNE_VERSION='"5.19.0-1027-gcp"' -I../include -I/usr/include -I/usr/include/libnl3 -I../include/uapi bpftune.o -o bpftune \ -L. -L/usr/local/lib64 -lbpf -ldl -lm -lrt -lcap -lnl-3 -lpthread -lnl-route-3 -lbpftune /usr/bin/ld: ./libbpftune.so: undefined reference to `bpf_object__next_program' collect2: error: ld returned 1 exit status make[1]: [Makefile:137: bpftune] Error 1 make[1]: Leaving directory '/usr/src/bpf/bpftune/src' make: [Makefile:41: srcdir] Error 2

[alan-maguire]

hmm, bpf_object__next_program() should be in libbpf v0.6 (min version supported for bpftune) and later; which version of libbpf are you using above?

[andrey-admin]

Seems Ubuntu 22.04.2 LTS still have old version:

dpkg -l |grep libbpf

ii libbpf-dev:amd64 1:0.5.0-1ubuntu22.04.1 amd64 eBPF helper library (development files) ii libbpf0:amd64 1:0.5.0-1ubuntu22.04.1 amd64 eBPF helper library (shared library) ii libbpfcc 0.18.0+ds-2 amd64 shared library for BPF Compiler Collection (BCC) ii libbpfcc-dev 0.18.0+ds-2 amd64 shared library for BPF Compiler Collection (BCC)

any suggestions?

[alan-maguire]

hmm, can you pull again and try? i've switch from using bpf_object__for_each_program() to iterating through the per-strategy list of programs and using bpf_object__find_program_by_name() which is available in libbpf 0.5

[andrey-admin]

build successful, checking in work.

thanks!

[andrey-admin]

is that ok?

Jul 24 12:26:49 nginx-27 bpftune[38277]: user modified sysctl 'ipv4/tcp_wmem' that tuner 'tcp_buffer' uses; disabling 'tcp_buffer' for namespace cookie 1 Jul 24 12:26:49 nginx-27 bpftune[38277]: user modified sysctl 'ipv4/tcp_rmem' that tuner 'tcp_buffer' uses; disabling 'tcp_buffer' for namespace cookie 1 Jul 24 12:26:49 nginx-27 bpftune[38277]: user modified sysctl 'default/gc_thresh3' that tuner 'neigh_table' uses; disabling 'neigh_table' for namespace cookie 1 Jul 24 12:26:49 nginx-27 bpftune[38277]: user modified sysctl 'default/gc_thresh2' that tuner 'neigh_table' uses; disabling 'neigh_table' for namespace cookie 1 Jul 24 12:26:49 nginx-27 bpftune[38277]: user modified sysctl 'default/gc_thresh1' that tuner 'neigh_table' uses; disabling 'neigh_table' for namespace cookie 1 Jul 24 12:26:49 nginx-27 bpftune[38277]: user modified sysctl 'default/gc_stale_time' that tuner 'neigh_table' uses; disabling 'neigh_table' for namespace cookie 1 Jul 24 12:26:49 nginx-27 bpftune[38277]: user modified sysctl 'ipv4/tcp_max_orphans' that tuner 'tcp_buffer' uses; disabling 'tcp_buffer' for namespace cookie 1 Jul 24 12:26:49 nginx-27 bpftune[38277]: user modified sysctl 'core/netdev_max_backlog' that tuner 'net_buffer' uses; disabling 'net_buffer' for namespace cookie 1 Jul 24 12:27:17 nginx-27 bpftune[38277]: ring_buffer__poll: Interrupted system call

[alan-maguire]

the above shows that bpftune has detected nginx applying its own tunable settings; in response it disables the tuners that auto-tune the same settings, to avoid bpftune's auto-tuning colliding with administrator intent.

[andrey-admin]

nginx-27 - that is node hostname, nginx do not do any tunes.

[alan-maguire]

i've just merged a change that will log the pid of the process outside of bpfevent changing the sysctls; if you rebuild and run that should tell you where the changes are coming from; messages are of form

bpftune: user (pid 8940) modified sysctl 'default/gc_thresh1' that tuner 'neigh_table' uses; disabling 'neigh_table' for namespace cookie 1

[andrey-admin]

rebuild done, checking in work.

[andrey-admin]

seem messages like:

modified sysctl 'default/gc_stale_time' that tuner 'neigh_table' uses; disabling 'neigh_table' for namespace cookie 1

appears when sysctl -p used after bpftune started. may be possible add adaptive behavior for that?

[alan-maguire]

that behaviour is intended; the idea is that the bpftune service runs after sysctl modifications at boot time, and if the admin applies any others that overlap with bpftune operation we disable the associated tuners. that said, at some point we'll probably have to add a bpftune.conf file for options like disabling specific tuners, etc. if the sysctl tuner were disabled, this behaviour wouldn't occur. so one approach would be to file an issue for support for bpftune.conf to allow users to customize bpftune/tuner config. i'd suggest we create a separate issue for that and close this one since the build issues appear to be resolved.