search

ajor / bpftrace

High-level tracing language for Linux eBPF - development moved to https://github.com/iovisor/bpftrace
https://github.com/iovisor/bpftrace
Apache License 2.0
250 stars 15 forks source link

bpf: Failed to load program: Bad file descriptor, using maps #9

Closed brendangregg closed 6 years ago

brendangregg commented 6 years ago

After fixing kern_version (#8), this Ubuntu Linux 4.4.90 system can then do printf(), but not maps. Eg:

# strace -febpf ./src/bpftrace -e 'kprobe:sys_read { @who[tid] = count(); }'
bpf(BPF_MAP_CREATE, {map_type=0x5 /* BPF_MAP_TYPE_??? */, key_size=8, value_size=8, max_entries=128}, 72) = -1 EINVAL (Invalid argument)
bpf(BPF_MAP_CREATE, {map_type=0x5 /* BPF_MAP_TYPE_??? */, key_size=8, value_size=8, max_entries=128}, 72) = -1 EINVAL (Invalid argument)
Error creating map: '@who'
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=8}, 72) = -1 EINVAL (Invalid argument)
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=8}, 72) = 3
Attaching 1 probe...
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7ffd24fc56b0, value=0x7ffd24fc56b4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7ffd24fc56b0, value=0x7ffd24fc56b4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7ffd24fc56b0, value=0x7ffd24fc56b4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7ffd24fc56b0, value=0x7ffd24fc56b4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7ffd24fc56b0, value=0x7ffd24fc56b4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7ffd24fc56b0, value=0x7ffd24fc56b4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7ffd24fc56b0, value=0x7ffd24fc56b4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7ffd24fc56b0, value=0x7ffd24fc56b4, flags=BPF_ANY}, 72) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=24, insns=0x7f864dd83000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=263258}, 72) = -1 E2BIG (Argument list too long)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=24, insns=0x7f864dd83000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=263258}, 72) = -1 EPERM (Operation not permitted)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=24, insns=0x7f864dd83000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=263258}, 72) = -1 EBADF (Bad file descriptor)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=24, insns=0x7f864dd83000, license="GPL", log_level=1, log_size=65536, log_buf=0x349ec10, kern_version=263258}, 72) = -1 EBADF (Bad file descriptor)
bpf: Failed to load program: Bad file descriptor
fd -1 is not pointing to valid bpf_map

Error loading program: kprobe:sys_read
+++ exited with 255 +++

This works fine on a 4.14 system:

# ./src/bpftrace -e 'kprobe:sys_read { @who[tid] = count(); }'
Attaching 1 probe...
^C

@who[2060]: 1
@who[25480]: 2
@who[1947]: 2
@who[1]: 3
@who[1830]: 14
@who[2418]: 30

I wonder if it's the same as issue #8 somehow.

brendangregg commented 6 years ago

EDIT: ignore this comment, it's a separate issue I've refiled as #18.

Perhaps related perhaps not, comm also doesn't work on that 4.4.90 Ubuntu system:

# strace -febpf ./src/bpftrace -e 'kprobe:sys_nanosleep { printf("sleep by %s\n", comm); }'
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=8}, 72) = -1 EINVAL (Invalid argument)
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=8}, 72) = 3
Attaching 1 probe...
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7fff19f71bb0, value=0x7fff19f71bb4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7fff19f71bb0, value=0x7fff19f71bb4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7fff19f71bb0, value=0x7fff19f71bb4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7fff19f71bb0, value=0x7fff19f71bb4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7fff19f71bb0, value=0x7fff19f71bb4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7fff19f71bb0, value=0x7fff19f71bb4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7fff19f71bb0, value=0x7fff19f71bb4, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x7fff19f71bb0, value=0x7fff19f71bb4, flags=BPF_ANY}, 72) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=257, insns=0x7f5c60865000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=263258}, 72) = -1 E2BIG (Argument list too long)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=257, insns=0x7f5c60865000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=263258}, 72) = -1 EPERM (Operation not permitted)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=257, insns=0x7f5c60865000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=263258}, 72) = -1 EACCES (Permission denied)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=257, insns=0x7f5c60865000, license="GPL", log_level=1, log_size=65536, log_buf=0x3489630, kern_version=263258}, 72) = -1 EACCES (Permission denied)
bpf: Failed to load program: Permission denied
0: (7b) *(u64 *)(r10 -192) = r1
1: (b7) r1 = 0
2: (7b) *(u64 *)(r10 -136) = r1
3: (bf) r1 = r10
4: (07) r1 += -64
5: (b7) r2 = 64
6: (85) call 16
invalid indirect read from stack off -64+0 size 64

Error loading program: kprobe:sys_nanosleep
+++ exited with 255 +++
brendangregg commented 6 years ago

The first bug was visible in the strace output:

map_type=0x5 /* BPF_MAP_TYPE_??? */

So that's BPF_MAP_TYPE_PERCPU_HASH, but this Linux 4.4 system doesn't have that -- that was added in 4.6 I think. Switching that to BPF_MAP_TYPE_HASH makes it work. Maybe we need a Linux version test.

taem commented 6 years ago

Works on 4.15.4:

$ sudo strace -febpf ./bpftrace -e 'kprobe:sys_read { @who[comm] = count(); }'
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_HASH, key_size=64, value_size=8, max_entries=128, map_flags=0, inner_map_fd=0, ...}, 72) = -1 EINVAL (Invalid argument)
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_HASH, key_size=64, value_size=8, max_entries=128, map_flags=0, inner_map_fd=0}, 72) = 3
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=2, map_flags=0, inner_map_fd=0, ...}, 72) = 4
Attaching 1 probe...
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=4, key=0x7fff7ae05c2c, value=0x7fff7ae05c08, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=4, key=0x7fff7ae05c2c, value=0x7fff7ae05c08, flags=BPF_ANY}, 72) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=262, insns=0x7fdbd7069000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=265988, prog_flags=0, ...}, 72) = -1 EINVAL (Invalid argument)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=262, insns=0x7fdbd7069000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=265988, prog_flags=0}, 72) = 8
^Cstrace: Process 16120 detached

@who[kuiserver5]: 1
@who[klauncher]: 1
@who[kscreen_backend]: 1
[ ... ]
JorgenRask commented 6 years ago

Doesn't work on Ubuntu 17.10 (kernel 4.13) either:

# strace -febpf ./bpftrace -e 'kprobe:sys_read { @who[comm] = count(); }'
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_HASH, key_size=64, value_size=8, max_entries=128}, 72) = -1 EINVAL (Invalid argument)
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_HASH, key_size=64, value_size=8, max_entries=128}, 72) = 3
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=4}, 72) = -1 EINVAL (Invalid argument)
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=4}, 72) = 4
Attaching 1 probe...
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=4, key=0x7fffef03ce80, value=0x7fffef03ce84, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=4, key=0x7fffef03ce80, value=0x7fffef03ce84, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=4, key=0x7fffef03ce80, value=0x7fffef03ce84, flags=BPF_ANY}, 72) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=4, key=0x7fffef03ce80, value=0x7fffef03ce84, flags=BPF_ANY}, 72) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=49, insns=0x7f6034f62000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=265472}, 72) = -1 E2BIG (Argument list too long)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=49, insns=0x7f6034f62000, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=265472}, 72) = -1 EINVAL (Invalid argument)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=49, insns=0x7f6034f62000, license="GPL", log_level=1, log_size=65536, log_buf=0x561553519740, kern_version=265472}, 72) = -1 EINVAL (Invalid argument)
bpf: Failed to load program: Invalid argument

Error loading program: kprobe:sys_read
+++ exited with 255 +++
# uname -a
Linux jr-ThinkPad-T430 4.13.0-37-generic #42-Ubuntu SMP Wed Mar 7 14:13:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

I also have problems with many other examples from #34

JorgenRask commented 6 years ago

I think above comment is more appropriate under issue #8. I have just tried patch #16 and it works for me.

ajor commented 6 years ago

Fixed with #17