Ubuntu: libbpftune.so.0.1.3: cannot open shared object file

[twirrim]

Tried a quick experiment on Ubuntu 23.04. Builds fine. On execution it throws up:

$ sudo bpftune 
bpftune: error while loading shared libraries: libbpftune.so.0.1.3: cannot open shared object file: No such file or directory

Locate shows that the library exists under /usr/lib64/libbpftune.so.0.1.3 strace output:

sudo strace bpftune
execve("/usr/sbin/bpftune", ["bpftune"], 0x7ffcb283c6c0 /* 31 vars */) = 0
brk(NULL)                               = 0x55b608e36000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd5b46b640) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa96949c000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=160489, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 160489, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fa969474000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3", 0x7ffd5b46a870, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2", 0x7ffd5b46a870, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/x86_64-linux-gnu", {st_mode=S_IFDIR|0755, st_size=249856, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3", 0x7ffd5b46a870, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2", 0x7ffd5b46a870, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu", {st_mode=S_IFDIR|0755, st_size=249856, ...}, 0) = 0
openat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v3/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v3", 0x7ffd5b46a870, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v2/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib/glibc-hwcaps/x86-64-v2", 0x7ffd5b46a870, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/lib", {st_mode=S_IFDIR|0755, st_size=16384, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v3/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v3", 0x7ffd5b46a870, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v2/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/glibc-hwcaps/x86-64-v2", 0x7ffd5b46a870, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/libbpftune.so.0.1.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib", {st_mode=S_IFDIR|0755, st_size=16384, ...}, 0) = 0
writev(2, [{iov_base="bpftune", iov_len=7}, {iov_base=": ", iov_len=2}, {iov_base="error while loading shared libra"..., iov_len=36}, {iov_base=": ", iov_len=2}, {iov_base="libbpftune.so.0.1.3", iov_len=19}, {iov_base=": ", iov_len=2}, {iov_base="cannot open shared object file", iov_len=30}, {iov_base=": ", iov_len=2}, {iov_base="No such file or directory", iov_len=25}, {iov_base="\n", iov_len=1}], 10bpftune: error while loading shared libraries: libbpftune.so.0.1.3: cannot open shared object file: No such file or directory
) = 126
exit_group(127)                         = ?
+++ exited with 127 +++

I can fudge around this by just putting a symlink for the library under /usr/lib to /usr/lib64:

sudo ln -snf /usr/lib64/libbpftune.so.0.1.3 /usr/lib/libbpftune.so.0.1.3

[twirrim]

Not exactly the same lines, but:

bpftune: error initializing '/usr/lib64/bpftune//route_table_tuner.so: No such file or directory

file exists:

$ stat /usr/lib64/bpftune/route_table_tuner.so
  File: /usr/lib64/bpftune/route_table_tuner.so
  Size: 1826464     Blocks: 3568       IO Block: 4096   regular file
...

maybe it's that extra trailing slash causing issues?

It does seem to be working for other tasks:

bpftune: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
bpftune: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 6291456) -> (4096 131072 7864320)
bpftune: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
bpftune: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 7864320) -> (4096 131072 9830400)
bpftune: Scenario 'specify bbr congestion control' occurred for tunable 'TCP congestion control' in global ns. Because loss rate has exceeded 1 percent for a connection, use bbr congestion control algorithm instead of default
bpftune: due to loss events for 130.35.24.119, specify 'bbr' congestion control algorithm
bpftune: Scenario 'specify bbr congestion control' occurred for tunable 'TCP congestion control' in global ns. Because loss rate has exceeded 1 percent for a connection, use bbr congestion control algorithm instead of default
bpftune: due to loss events for 10.44.105.53, specify 'bbr' congestion control algorithm
bpftune: Scenario 'netns created' occurred for tunable 'Network namespace' in non-global ns. network namespace creation
bpftune: netns created (cookie 12857)

[alan-maguire]

thanks for the report! i'll push a fix for the trailing slash issue though I don't think it should cause this. Would you mind attaching the output of "bpftune -ds" (runs in debug mode, output to stdout/stderr instead of syslog)? That should help to get to the bottom of the route tuner issues. It's possible the bpf attach target is missing or optimized out. thanks again!

[alan-maguire]

if ubuntu doesn't use /usr/lib64 we should probably fix up the Makefile/specfile to support an alternative location

[MmartinezV2]

Same problem on Debian 11. Solved with same symlink.

Two problems for build: 1) On file include/bpftune/bpftune.bpf.h fails because NULL is undefined. Solved by adding at the begining of the file:

ifndef NULL

define NULL (0)

endif

2) File bpf/libbpf_version.h does not exist Solved by commenting line 54 #include <bpf/libbpf_version.h> in file src/libbpftune.c

Output:

bpftune -S
bpftune: bpftune works fully
bpftune: bpftune does not support per-netns policy (via netns cookie)

service bpftune status
● bpftune.service - BPF-based auto-tuning of system parameters
     Loaded: loaded (/lib/systemd/system/bpftune.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-07-06 14:50:08 CEST; 20min ago
   Main PID: 18686 (bpftune)
      Tasks: 2 (limit: 9492)
     Memory: 7.9M
        CPU: 1.120s
     CGroup: /system.slice/bpftune.service
             └─18686 /usr/sbin/bpftune

jul 06 14:50:08 odb-prev systemd[1]: Started BPF-based auto-tuning of system parameters.
jul 06 14:50:08 odb-prev bpftune[18686]: bpftune works fully
jul 06 14:50:08 odb-prev bpftune[18686]: bpftune does not support per-netns policy (via netns cookie)
jul 06 14:50:09 odb-prev bpftune[18686]: could not load skeleton: Kernel verifier blocks program loading
jul 06 14:50:09 odb-prev bpftune[18686]: error initializing '/usr/lib64/bpftune//tcp_cong_tuner.so: Unknown error 4007
jul 06 14:50:09 odb-prev bpftune[18686]: error initializing '/usr/lib64/bpftune//netns_tuner.so: Operation not supported

[alan-maguire]

thanks for all the info! I've got a fix coming to address the library search issues, and a few other cleanups I found when testing on Ubuntu. I reckon dealing with the above debian-related problems in a separate issue might make sense; @MmartinezV2 I'll open that and credit you with a reported-by in the fix if that works?

[twirrim]

That PR seems to have fixed the main library path issue (cleared symlink, all looks good). It's still complaining about:

bpftune: could not open /proc/sys/net/ipv6/route/max_size (netns fd 0) for reading: No such file or directory
bpftune: error reading tunable 'net.ipv6.route.max_size': No such file or directory
bpftune: error initializing '/usr/lib64/bpftune//route_table_tuner.so: No such file or directory
bpftune: could not open /proc/sys/net/ipv6/neigh/default/gc_interval (netns fd 0) for reading: No such file or directory
bpftune: error reading tunable 'net.ipv6.neigh.default.gc_interval': No such file or directory
bpftune: error initializing '/usr/lib64/bpftune//neigh_table_tuner.so: No such file or directory

note: I've got IPv6 hard disabled, I'm on network that really doesn't behave well with IPv6 for reasons outside of my control. I assume that's why the tunable messages, that's fine. Not sure if that's behind the library messages. Those files definitely exist. My gut feeling is it probably is. I've attached the debug output. I'll try and re-enable IPv6 later (been a while, maybe they've fixed the issues) and give it another shot.

bpftunelogs.txt

[twirrim]

Confirmed those bpftune: error initializing '/usr/lib64/bpftune//neigh_table_tuner.so: No such file or directory were happening because I had ipv6.disable=1 as a kernel parameter.

[alan-maguire]

thanks for confirming. It wouldn't be unreasonable to have better handling for the ipv6 disabled case in bpftune - tuners should still handle ipv4 in such cases - so feel free to file a separate issue for that.