btf: CO-RE against kernel module fails with invalid relocation

[bitness]

Hello, I'm having trouble using the kernel's container_of macro in a BPF program loaded with cilium/ebpf v0.10.0. This program works as expected when loaded with libbpf 0.6.0 or by running sudo bpftool prog load rmdir-watch.bpf.o /sys/fs/bpf/rmdir-watch.

I'll attach a zip file with all the files needed to reproduce it, but to summarize, the BPF program tries to do this:

static inline struct nfs_inode* NFS_I(const struct inode* inode)
{
    return container_of(inode, struct nfs_inode, vfs_inode);
}

SEC("fexit/nfs_rmdir")
int BPF_PROG(nfs_rmdir, struct inode* dir, struct dentry* dentry, int ret)
{
    struct nfs_inode* nfsi = NFS_I(dir);
    if !(nfsi)
        return 0;
    /* irrelevant stuff */
    return 0;
}

container_of does some pointer math:

#ifndef offsetof
#define offsetof(TYPE, MEMBER)  ((unsigned long)&((TYPE *)0)->MEMBER)
#endif
#ifndef container_of
#define container_of(ptr, type, member)                         \
        ({                                                      \
                void *__mptr = (void *)(ptr);                   \
                ((type *)(__mptr - offsetof(type, member)));    \
        })
#endif

When I run the go code using cilium/ebpf's bpf20 and calling loadBpfObjects to load the BPF program, it fails with this:

verifier error: load program: invalid argument:
        func#0 @0
        R1 type=ctx expected=fp
        0: R1=ctx(id=0,off=0,imm=0) R10=fp0
        ; int BPF_PROG(nfs_rmdir, struct inode* dir, struct dentry* dentry, int ret)
        0: (85) call unknown#195896080
        invalid func unknown#195896080
        processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
2023/04/27 14:29:36 loading objects: field NfsRmdir: program nfs_rmdir: load program: invalid argument: invalid func unknown#195896080 (6 line(s) omitted)

Some details about the machine I'm developing on:

go version go1.18 linux/amd64
Linux el9-ws2 5.14.0-162.22.2.px3.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 5 20:48:25 PDT 2023 x86_64 x86_64 x86_64 GNU/Linux
AlmaLinux 9.1 (Lime Lynx)

When I run the same BPF program through libbpf 0.6.0, I'm able to write a simple C program that logs some of the struct nfs_inode's data whenever a directory is removed on an NFS mount. If the BPF program is working in that context, or through bpftool prog load, should I expect that it'd also work with cilium/ebpf v0.10.0 without changes?

In the attached zip file, the justfile has the commands used for building the Go test program and the C program. If the Go program loads the BPF program without error, it'll just print "it worked!" and exit; currently it prints the error above. The C program will succeed and log NFS rmdir calls.

Thanks for any advice! container-of-repro.zip

[lmb]

For some reason your program is causing our CO-RE code to bail out. 195896080 is 0xbad2310 in hex, which is https://github.com/cilium/ebpf/blob/4927ec860f48ed564beb410ffff2575cfbb2d98b/btf/core.go#L44 So most likely the cause is that there is a slight mismatch between libbpf and our rules around type eqality for CO-RE.

Can you please upload the compiled BPF .o? Compiling it on my machine doesn't work out of the box.

[bitness]

Certainly, here are the compiled object files. I'm including the ones from the go build as well as the one I built for the libbpf program, which was compiled with this command:

clang -g -O2 -target bpf -march=probe -D__TARGET_ARCH_x86 -Ix86/ -c rmdir-watch.bpf.c -o rmdir-watch.bpf.o

I'm doing the build on an AlmaLinux 9.1 machine with the clang-14.0.6-4.el9_1.x86_64 RPM. Here's its version output:

clang version 14.0.6 (Red Hat 14.0.6-4.el9_1)
Target: x86_64-redhat-linux-gnu
Thread model: posix
InstalledDir: /bin

The libbpf RPM is libbpf-0.6.0-1.el9.x86_64. Its source rpm is here if that's helpful.

container-of-repro-object-files.zip

[bitness]

Also, if it helps--even with libbpf, I've found that I have to use bpf_core_read and friends to access data through pointers I get out of container_of usage. I think the pointer math it does is weird enough to confuse libbpf's automatic access conversions.

[lmb]

So, the problem is that struct nfs_inode is not part of vmlinux BTF, instead it is part of module nfs. Our current CO-RE code is oblivious to kernel modules, and so can't find nfs_inode. We don't bail out with an error, since it's valid to reference non-existent types when their use is guarded by if core_type_exists(...).

So, unfortunately, you've run into the last remaining point of #705. I'm also unhappy that the error message is so obtuse, unfortunately the error is output by the kernel. Maybe there is a way to improve that.

[bitness]

Thanks for tracking this down! I subscribed to #705 and will try again once it's resolved.

[lmb]

This should be doable now.