iovisor / bcc

BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more
Apache License 2.0
20.64k stars 3.89k forks source link

libbpf-tools: 'mount_entry': failed to attach to perf_event FD 12: Permission denied #4068

Open setNull opened 2 years ago

setNull commented 2 years ago

Hi team,

I'm trying to run mountsnoop on arm64 target, but failed as below,

root@tegra-ubuntu:/ota/test# ./mountsnoop -v
libbpf: loading object 'mountsnoop_bpf' from buffer
libbpf: elf: section(1) .text, size 960, link 0, flags 6, type=1
libbpf: sec '.text': found program 'probe_exit' at insn offset 0 (0 bytes), code size 120 insns (960 bytes)
libbpf: elf: section(2) tracepoint/syscalls/sys_enter_mount, size 352, link 0, flags 6, type=1
libbpf: sec 'tracepoint/syscalls/sys_enter_mount': found program 'mount_entry' at insn offset 0 (0 bytes), code size 44 insns (352 bytes)
libbpf: elf: section(3) tracepoint/syscalls/sys_exit_mount, size 32, link 0, flags 6, type=1
libbpf: sec 'tracepoint/syscalls/sys_exit_mount': found program 'mount_exit' at insn offset 0 (0 bytes), code size 4 insns (32 bytes)
libbpf: elf: section(4) tracepoint/syscalls/sys_enter_umount, size 304, link 0, flags 6, type=1
libbpf: sec 'tracepoint/syscalls/sys_enter_umount': found program 'umount_entry' at insn offset 0 (0 bytes), code size 38 insns (304 bytes)
libbpf: elf: section(5) tracepoint/syscalls/sys_exit_umount, size 32, link 0, flags 6, type=1
libbpf: sec 'tracepoint/syscalls/sys_exit_umount': found program 'umount_exit' at insn offset 0 (0 bytes), code size 4 insns (32 bytes)
libbpf: elf: section(6) .rodata, size 4, link 0, flags 2, type=1
libbpf: elf: section(7) license, size 13, link 0, flags 3, type=1
libbpf: license of mountsnoop_bpf is Dual BSD/GPL
libbpf: elf: section(8) .maps, size 88, link 0, flags 3, type=1
libbpf: elf: section(9) .BTF, size 26557, link 0, flags 0, type=1
libbpf: elf: section(10) .BTF.ext, size 2076, link 0, flags 0, type=1
libbpf: elf: section(11) .symtab, size 720, link 20, flags 0, type=2
libbpf: elf: section(12) .rel.text, size 64, link 11, flags 0, type=9
libbpf: elf: section(13) .reltracepoint/syscalls/sys_enter_mount, size 32, link 11, flags 0, type=9
libbpf: elf: section(14) .reltracepoint/syscalls/sys_exit_mount, size 16, link 11, flags 0, type=9
libbpf: elf: section(15) .reltracepoint/syscalls/sys_enter_umount, size 32, link 11, flags 0, type=9
libbpf: elf: section(16) .reltracepoint/syscalls/sys_exit_umount, size 16, link 11, flags 0, type=9
libbpf: looking for externs among 30 symbols...
libbpf: collected 0 externs total
libbpf: map 'args': at sec_idx 8, offset 0.
libbpf: map 'args': found type = 1.
libbpf: map 'args': found key [9], sz = 4.
libbpf: map 'args': found value [12], sz = 56.
libbpf: map 'args': found max_entries = 10240.
libbpf: map 'heap': at sec_idx 8, offset 32.
libbpf: map 'heap': found type = 6.
libbpf: map 'heap': found key [3], sz = 4.
libbpf: map 'heap': found value [25], sz = 8768.
libbpf: map 'heap': found max_entries = 1.
libbpf: map 'events': at sec_idx 8, offset 64.
libbpf: map 'events': found type = 4.
libbpf: map 'events': found key_size = 4.
libbpf: map 'events': found value_size = 4.
libbpf: map 'mountsno.rodata' (global data): at sec_idx 6, offset 0, flags 480.
libbpf: map 3 is "mountsno.rodata"
libbpf: sec '.rel.text': collecting relocation for section(1) '.text'
libbpf: sec '.rel.text': relo #0: insn #9 against 'args'
libbpf: prog 'probe_exit': found map 0 (args, sec 8, off 0) for insn #9
libbpf: sec '.rel.text': relo #1: insn #16 against 'heap'
libbpf: prog 'probe_exit': found map 1 (heap, sec 8, off 32) for insn #16
libbpf: sec '.rel.text': relo #2: insn #107 against 'events'
libbpf: prog 'probe_exit': found map 2 (events, sec 8, off 64) for insn #107
libbpf: sec '.rel.text': relo #3: insn #116 against 'args'
libbpf: prog 'probe_exit': found map 0 (args, sec 8, off 0) for insn #116
libbpf: sec '.reltracepoint/syscalls/sys_enter_mount': collecting relocation for section(2) 'tracepoint/syscalls/sys_enter_mount'
libbpf: sec '.reltracepoint/syscalls/sys_enter_mount': relo #0: insn #17 against 'target_pid'
libbpf: prog 'mount_entry': found data map 3 (mountsno.rodata, sec 6, off 0) for insn 17
libbpf: sec '.reltracepoint/syscalls/sys_enter_mount': relo #1: insn #38 against 'args'
libbpf: prog 'mount_entry': found map 0 (args, sec 8, off 0) for insn #38
libbpf: sec '.reltracepoint/syscalls/sys_exit_mount': collecting relocation for section(3) 'tracepoint/syscalls/sys_exit_mount'
libbpf: sec '.reltracepoint/syscalls/sys_exit_mount': relo #0: insn #1 against '.text'
libbpf: sec '.reltracepoint/syscalls/sys_enter_umount': collecting relocation for section(4) 'tracepoint/syscalls/sys_enter_umount'
libbpf: sec '.reltracepoint/syscalls/sys_enter_umount': relo #0: insn #12 against 'target_pid'
libbpf: prog 'umount_entry': found data map 3 (mountsno.rodata, sec 6, off 0) for insn 12
libbpf: sec '.reltracepoint/syscalls/sys_enter_umount': relo #1: insn #32 against 'args'
libbpf: prog 'umount_entry': found map 0 (args, sec 8, off 0) for insn #32
libbpf: sec '.reltracepoint/syscalls/sys_exit_umount': collecting relocation for section(5) 'tracepoint/syscalls/sys_exit_umount'
libbpf: sec '.reltracepoint/syscalls/sys_exit_umount': relo #0: insn #1 against '.text'
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: map 'args': created successfully, fd=4
libbpf: map 'heap': created successfully, fd=5
libbpf: map 'events': setting size to 11
libbpf: map 'events': created successfully, fd=6
libbpf: map 'mountsno.rodata': created successfully, fd=7
libbpf: sec 'tracepoint/syscalls/sys_enter_mount': found 5 CO-RE relocations
libbpf: CO-RE relocating [36] struct trace_event_raw_sys_enter: found target candidate [5156] struct trace_event_raw_sys_enter in [vmlinux]
libbpf: prog 'mount_entry': relo #0: <byte_off> [36] struct trace_event_raw_sys_enter.args[4] (0:2:4 @ offset 48)
libbpf: prog 'mount_entry': relo #0: matching candidate #0 <byte_off> [5156] struct trace_event_raw_sys_enter.args[4] (0:2:4 @ offset 56)
libbpf: prog 'mount_entry': relo #0: patched insn #0 (LDX/ST/STX) off 48 -> 56
libbpf: prog 'mount_entry': relo #1: <byte_off> [36] struct trace_event_raw_sys_enter.args[3] (0:2:3 @ offset 40)
libbpf: prog 'mount_entry': relo #1: matching candidate #0 <byte_off> [5156] struct trace_event_raw_sys_enter.args[3] (0:2:3 @ offset 48)
libbpf: prog 'mount_entry': relo #1: patched insn #2 (LDX/ST/STX) off 40 -> 48
libbpf: prog 'mount_entry': relo #2: <byte_off> [36] struct trace_event_raw_sys_enter.args[2] (0:2:2 @ offset 32)
libbpf: prog 'mount_entry': relo #2: matching candidate #0 <byte_off> [5156] struct trace_event_raw_sys_enter.args[2] (0:2:2 @ offset 40)
libbpf: prog 'mount_entry': relo #2: patched insn #4 (LDX/ST/STX) off 32 -> 40
libbpf: prog 'mount_entry': relo #3: <byte_off> [36] struct trace_event_raw_sys_enter.args[1] (0:2:1 @ offset 24)
libbpf: prog 'mount_entry': relo #3: matching candidate #0 <byte_off> [5156] struct trace_event_raw_sys_enter.args[1] (0:2:1 @ offset 32)
libbpf: prog 'mount_entry': relo #3: patched insn #5 (LDX/ST/STX) off 24 -> 32
libbpf: prog 'mount_entry': relo #4: <byte_off> [36] struct trace_event_raw_sys_enter.args[0] (0:2:0 @ offset 16)
libbpf: prog 'mount_entry': relo #4: matching candidate #0 <byte_off> [5156] struct trace_event_raw_sys_enter.args[0] (0:2:0 @ offset 24)
libbpf: prog 'mount_entry': relo #4: patched insn #6 (LDX/ST/STX) off 16 -> 24
libbpf: sec 'tracepoint/syscalls/sys_exit_mount': found 1 CO-RE relocations
libbpf: CO-RE relocating [47] struct trace_event_raw_sys_exit: found target candidate [5157] struct trace_event_raw_sys_exit in [vmlinux]
libbpf: prog 'mount_exit': relo #0: <byte_off> [47] struct trace_event_raw_sys_exit.ret (0:2 @ offset 16)
libbpf: prog 'mount_exit': relo #0: matching candidate #0 <byte_off> [5157] struct trace_event_raw_sys_exit.ret (0:2 @ offset 24)
libbpf: prog 'mount_exit': relo #0: patched insn #0 (LDX/ST/STX) off 16 -> 24
libbpf: sec '.text': found 3 CO-RE relocations
libbpf: CO-RE relocating [53] struct task_struct: found target candidate [148] struct task_struct in [vmlinux]
libbpf: prog 'probe_exit': relo #0: <byte_off> [53] struct task_struct.nsproxy (0:92 @ offset 1672)
libbpf: prog 'probe_exit': relo #0: matching candidate #0 <byte_off> [148] struct task_struct.nsproxy (0:115 @ offset 1856)
libbpf: prog 'probe_exit': relo #0: patched insn #36 (ALU/ALU64) imm 1672 -> 1856
libbpf: CO-RE relocating [315] struct nsproxy: found target candidate [453] struct nsproxy in [vmlinux]
libbpf: prog 'probe_exit': relo #1: <byte_off> [315] struct nsproxy.mnt_ns (0:3 @ offset 24)
libbpf: prog 'probe_exit': relo #1: matching candidate #0 <byte_off> [453] struct nsproxy.mnt_ns (0:3 @ offset 24)
libbpf: prog 'probe_exit': relo #1: patched insn #46 (ALU/ALU64) imm 24 -> 24
libbpf: CO-RE relocating [321] struct mnt_namespace: found target candidate [27773] struct mnt_namespace in [vmlinux]
libbpf: prog 'probe_exit': relo #2: <byte_off> [321] struct mnt_namespace.ns.inum (0:1:2 @ offset 24)
libbpf: prog 'probe_exit': relo #2: matching candidate #0 <byte_off> [27773] struct mnt_namespace.ns.inum (0:1:2 @ offset 24)
libbpf: prog 'probe_exit': relo #2: patched insn #52 (ALU/ALU64) imm 24 -> 24
libbpf: sec 'tracepoint/syscalls/sys_enter_umount': found 2 CO-RE relocations
libbpf: prog 'umount_entry': relo #0: <byte_off> [36] struct trace_event_raw_sys_enter.args[1] (0:2:1 @ offset 24)
libbpf: prog 'umount_entry': relo #0: matching candidate #0 <byte_off> [5156] struct trace_event_raw_sys_enter.args[1] (0:2:1 @ offset 32)
libbpf: prog 'umount_entry': relo #0: patched insn #0 (LDX/ST/STX) off 24 -> 32
libbpf: prog 'umount_entry': relo #1: <byte_off> [36] struct trace_event_raw_sys_enter.args[0] (0:2:0 @ offset 16)
libbpf: prog 'umount_entry': relo #1: matching candidate #0 <byte_off> [5156] struct trace_event_raw_sys_enter.args[0] (0:2:0 @ offset 24)
libbpf: prog 'umount_entry': relo #1: patched insn #1 (LDX/ST/STX) off 16 -> 24
libbpf: sec 'tracepoint/syscalls/sys_exit_umount': found 1 CO-RE relocations
libbpf: prog 'umount_exit': relo #0: <byte_off> [47] struct trace_event_raw_sys_exit.ret (0:2 @ offset 16)
libbpf: prog 'umount_exit': relo #0: matching candidate #0 <byte_off> [5157] struct trace_event_raw_sys_exit.ret (0:2 @ offset 24)
libbpf: prog 'umount_exit': relo #0: patched insn #0 (LDX/ST/STX) off 16 -> 24
libbpf: prog 'mount_exit': added 120 insns from sub-prog 'probe_exit'
libbpf: prog 'mount_exit': insn #1 relocated, imm 2 points to subprog 'probe_exit' (now at 4 offset)
libbpf: prog 'umount_exit': added 120 insns from sub-prog 'probe_exit'
libbpf: prog 'umount_exit': insn #1 relocated, imm 2 points to subprog 'probe_exit' (now at 4 offset)
libbpf: prog 'mount_entry': failed to attach to perf_event FD 12: Permission denied
libbpf: prog 'mount_entry': failed to attach to tracepoint 'syscalls/sys_enter_mount': Permission denied
libbpf: prog 'mount_entry': failed to auto-attach: -13
failed to attach BPF programs: -13

How should I get start with this error? Thanks.

chenhengqi commented 2 years ago

Could you check whether syscalls/sys_enter_mount tracepoint exists ?

Rtoax commented 1 year ago

I got the same problem, This could be an issue with different OS's kernel version.

CentOS-Stream-9 for example:

$ sudo ./mountsnoop
libbpf: prog 'mount_entry': failed to create BPF link for perf_event FD 11: -13 (Permission denied)
libbpf: prog 'mount_entry': failed to attach to tracepoint 'syscalls/sys_enter_mount': Permission denied
libbpf: prog 'mount_entry': failed to auto-attach: -13
failed to attach BPF programs: -13

$ sudo ./execsnoop
libbpf: prog 'tracepoint__syscalls__sys_exit_execve': failed to create BPF link for perf_event FD 12: -13 (Permission denied)
libbpf: prog 'tracepoint__syscalls__sys_exit_execve': failed to attach to tracepoint 'syscalls/sys_exit_execve': Permission denied
libbpf: prog 'tracepoint__syscalls__sys_exit_execve': failed to auto-attach: -13
failed to attach BPF programs

$ sudo ./opensnoop
libbpf: prog 'tracepoint__syscalls__sys_exit_open': failed to create BPF link for perf_event FD 15: -13 (Permission denied)
libbpf: prog 'tracepoint__syscalls__sys_exit_open': failed to attach to tracepoint 'syscalls/sys_exit_open': Permission denied
libbpf: prog 'tracepoint__syscalls__sys_exit_open': failed to auto-attach: -13
failed to attach BPF programs

The tracepoint exist.

$ sudo bpftrace -lv tracepoint:syscalls:sys_enter_mount
tracepoint:syscalls:sys_enter_mount
    int __syscall_nr
    char * dev_name
    char * dir_name
    char * type
    unsigned long flags
    void * data
setNull commented 1 year ago

Hi @chenhengqi , I revisit here...found this issue may be related to force type casting in bpf progs on arm64 platform. f.ex., a few changes like below world make it work,here "ctx->args[1]" is int,while "flags" is unsigned long

@@ -113,7 +113,7 @@ SEC("tracepoint/syscalls/sys_enter_umount")
 int umount_entry(struct trace_event_raw_sys_enter *ctx)
 {
        const char *dest = (const char *)ctx->args[0];
-       __u64 flags = (__u64)ctx->args[1];
+       __u64 flags = 0;//(__u64)ctx->args[1];

        return probe_entry(NULL, dest, NULL, flags, NULL, UMOUNT);
 }

Still not work out the reason, any idea? Thanks.

setNull commented 1 year ago

Hi @chenhengqi , more info... I found the failure is at https://github.com/torvalds/linux/blob/master/kernel/events/core.c#L10490 , at where prog->aux->max_ctx_offset is bigger than return value of trace_event_get_offsets in arm64, 5.10.120 kernel.

I'm still trying to understand how this issue happen. However, if I change code to visit ctx using BPF_CORE_READ macro, it works well. several changes like below,

 SEC("tracepoint/syscalls/sys_enter_umount")
 int umount_entry(struct trace_event_raw_sys_enter *ctx)
 {
        const char *dest = (const char *)ctx->args[0];
-       __u64 flags = (__u64)ctx->args[1];
+       __u64 flags = (__u64)BPF_CORE_READ(ctx, args[1]);

        return probe_entry(NULL, dest, NULL, flags, NULL, UMOUNT);
 }

Does this fix make sense for you?

setNull commented 1 year ago

Hi @chenhengqi , more info... I found the failure is at https://github.com/torvalds/linux/blob/master/kernel/events/core.c#L10490 , at where prog->aux->max_ctx_offset is bigger than return value of trace_event_get_offsets in arm64, 5.10.120 kernel.

I'm still trying to understand how this issue happen. However, if I change code to visit ctx using BPF_CORE_READ macro, it works well. several changes like below,

 SEC("tracepoint/syscalls/sys_enter_umount")
 int umount_entry(struct trace_event_raw_sys_enter *ctx)
 {
        const char *dest = (const char *)ctx->args[0];
-       __u64 flags = (__u64)ctx->args[1];
+       __u64 flags = (__u64)BPF_CORE_READ(ctx, args[1]);

        return probe_entry(NULL, dest, NULL, flags, NULL, UMOUNT);
 }

Does this fix make sense for you?

Hi @yonghong-song @chenhengqi , please look into this, should I make a PR fot this?

chenhengqi commented 1 year ago

@setNull Thanks for the details. What's your kernel version ? I can't reproduce locally.

setNull commented 1 year ago

@setNull Thanks for the details. What's your kernel version ? I can't reproduce locally.

5.10.120 arm64

setNull commented 1 year ago

@chenhengqi