Error: failed to pin program kretprobe/__x64_sys_time

[riyajha2305]

I actually wanted to ask how do I unload / detach a bpf prog.

I had initially loaded and attached a bpf program using

sudo bpftool prog load inject_error_v2.o /sys/fs/bpf/inject_error_v2 autoattach

and then unloaded it using

sudo rm /sys/fs/bpf/inject_error_v2 .

But now when I am trying to load it again, I get this error:

Error: failed to pin program kretprobe/__x64_sys_time

I tried doing sudo rm /sys/fs/bpf/inject_error_v2 again but got this:

rm: cannot remove '/sys/fs/bpf/inject_error_v2': No such file or directory

Also, on sudo ls /sys/fs/bpf , I get: maps.debug progs.debug

On doing sudo strace bpftool prog load inject_error_v2.o /sys/fs/bpf/inject_error_v2 autoattach

Not sure if the highlighted part is the cause of this error.

BPF_PROG_LOAD works and it returns FD=5 Before that the perf_event is created at FD=4 Then bpf(0x1c strace doesn't understand it but this is the BPF_LINK_CREATE command which fails. The attach logic then does the ioctls to attach which does work. But then at the end, it tries to BPF_OBJ_PIN but with bpf_fd=4 which is the perf_event. You can't pin a perf event, only links.

So it seems like 2 things.

1. For some reason the BPF link can't be created
2. There seems to be a bug in this version of BPF tool where it then tries to pin the wrong resource.

[qmonnet]

Thanks for the report! What version of bpftool are you running, please?

[qmonnet]

It might also be helpful to specify your kernel version.

I can't reproduce on my side. For what it's worth:

• maps.debug and progs.debug are BPF iterators that can be used for debugging purposes, preloaded into your kernel at boot time. They're not related to the program you're trying to load/unload.
• The error on the mkdir() is not an issue. It's libbpf trying to create the parent directory for the location where you try to pin your link, but it ignores EEXIST errors.
• I think the BPF link can be created. It's true that bpf(BPF_LINK_CREATE, ...) fails, but that one is likely a probe, and I believe libbpf falls back to an alternate way of pinning the link, via the ioctl(..., PERF_EVENT_IOC_SET_BPF, ...).

Now the question remains to understand why you can create and pin a link to your program on the first try, but not on a second attempt. How does it work if you try to load another program? Or the same program, but to pin it with a different path? And how does the strace output differs between a successful and a failed loads?

[riyajha2305]

It might also be helpful to specify your kernel version.

I can't reproduce on my side. For what it's worth:

• maps.debug and progs.debug are BPF iterators that can be used for debugging purposes, preloaded into your kernel at boot time. They're not related to the program you're trying to load/unload.
• The error on the mkdir() is not an issue. It's libbpf trying to create the parent directory for the location where you try to pin your link, but it ignores EEXIST errors.
• I think the BPF link can be created. It's true that bpf(BPF_LINK_CREATE, ...) fails, but that one is likely a probe, and I believe libbpf falls back to an alternate way of pinning the link, via the ioctl(..., PERF_EVENT_IOC_SET_BPF, ...).

Now the question remains to understand why you can create and pin a link to your program on the first try, but not on a second attempt. How does it work if you try to load another program? Or the same program, but to pin it with a different path? And how does the strace output differs between a successful and a failed loads?

bpftool version: v7.4.0-amd64

kernel version: 5.10.224-190.876.amzn2int.x86_64 (custom build of Amazon Linux 2 Kernel)

I realized that the program was initially getting loaded only as I was able to execute the load command successfully without autoattach

sudo bpftool prog load inject_error_v2.o /sys/fs/bpf/inject_error_v2

Hence, at that time I was able to see the loaded program on

sudo bpftool prog show

but the program was not seen while executing

sudo bpftool link

I would want to use the autoattach while loading as I believe bpftool supports autoattach option only, for kprobes / kretprobes type bpf programs instead of manual attach.

[qmonnet]

Right, without autoattach, the program is loaded and pinned, but not attached, so it does not run. It shows, with its pinned path, under bpftool prog list -f. Indeed, there is no support in bpftool, currently, for attaching a tracing program that was loaded with a previous invocation, the only way is to do it all in one go via autoattach.

With autoattach, it's automatically attached after being loaded, and the new BPF link is pinned. So you won't see the pinned path with bpftool prog list -f, but you will see the pinned link with bpftool link list -f.

If I follow correctly, you managed to load the program without autoattach the first time, then removed it, and tried with autoattach but didn't succeed, is that correct? Do you have a minimal program that I could try to reproduce the error with? Do you get the same results with something as minimal as this?:

#include <linux/bpf.h>
#include "bpf_helpers.h"

SEC("kprobe/__set_task_comm")
int foo(void)
{
    return 0;
}

[riyajha2305]

Right, without autoattach, the program is loaded and pinned, but not attached, so it does not run. It shows, with its pinned path, under bpftool prog list -f. Indeed, there is no support in bpftool, currently, for attaching a tracing program that was loaded with a previous invocation, the only way is to do it all in one go via autoattach.

With autoattach, it's automatically attached after being loaded, and the new BPF link is pinned. So you won't see the pinned path with bpftool prog list -f, but you will see the pinned link with bpftool link list -f.

If I follow correctly, you managed to load the program without autoattach the first time, then removed it, and tried with autoattach but didn't succeed, is that correct? Do you have a minimal program that I could try to reproduce the error with? Do you get the same results with something as minimal as this?:

#include <linux/bpf.h>
#include "bpf_helpers.h"

SEC("kprobe/__set_task_comm")
int foo(void)
{
    return 0;
}

yes you are right. I tried to load the program without autoattach the first time, then removed it, and tried with autoattach but didn't succeed.

I'm trying to load and attach this bpf program.

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

SEC("kretprobe/__x64_sys_syslog")
int bpf_prog_v2(struct pt_regs *ctx){
    long rc = -12;
    bpf_override_return(ctx, rc);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

[riyajha2305]

Right, without autoattach, the program is loaded and pinned, but not attached, so it does not run. It shows, with its pinned path, under bpftool prog list -f. Indeed, there is no support in bpftool, currently, for attaching a tracing program that was loaded with a previous invocation, the only way is to do it all in one go via autoattach.

With autoattach, it's automatically attached after being loaded, and the new BPF link is pinned. So you won't see the pinned path with bpftool prog list -f, but you will see the pinned link with bpftool link list -f.

If I follow correctly, you managed to load the program without autoattach the first time, then removed it, and tried with autoattach but didn't succeed, is that correct? Do you have a minimal program that I could try to reproduce the error with? Do you get the same results with something as minimal as this?:

#include <linux/bpf.h>
#include "bpf_helpers.h"

SEC("kprobe/__set_task_comm")
int foo(void)
{
    return 0;
}

output for the given program.

[riyajha2305]

Also, for your reference,I have enabled the following configs.

CONFIG_MODULES=y CONFIG_MODULE_UNLOAD=y CONFIG_FTRACE=y CONFIG_PERF_EVENTS=y CONFIG_KPROBES=y CONFIG_KRETPROBES=y CONFIG_KPROBE_EVENTS=y CONFIG_TRACEPOINTS=y CONFIG_BPF_SYSCALL=y CONFIG_BPF_EVENTS=y CONFIG_FUNCTION_ERROR_INJECTION=y CONFIG_BPF_KPROBE_OVERRIDE=y

[qmonnet]

OK, your program loads and attaches correctly on my setup (Linux 6.10), so there's something going wrong with the creation or pinning of the link, it seems. I'd need to set up a VM with a 5.10 kernel to try to reproduce, but I'm not sure when I can find the time. If you want to take a look, it might be interesting to run the command with retsnoop, to see if you can get more info on what function fails in the kernel.

If you need this fixed quickly, you can also send a report to the BPF mailing list (which should be the primary forum to report bpftool bugs), where there will be more eyes to take a look.