powerpc: libbpf: failed to find BTF for extern 'PT_REGS_PARM1': -2

[vt-alt]

(I try to package libbpf-tools from bcc for ALT Linux.) On ppc64le and aarch64 architectures libbpf-tools fail to compile in GEN-SKEL, because of bpftool error. Actual cause seems to be the clang compilation warnings shown below:

builder@powerpc64le:~/RPM/BUILD/bcc-0.18.0/libbpf-tools$ clang -g -O2 -target bpf -D__TARGET_ARCH_ppc64le -I.output -c biosnoop.bpf.c -o .output/biosnoop.bpf.o

biosnoop.bpf.c:65:5: warning: implicit declaration of function 'PT_REGS_PARM1' is invalid in C99 [-Wimplicit-function-declaration]
int BPF_KPROBE(blk_account_io_merge_bio, struct request *rq)
    ^
.output/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^
.output/bpf/bpf_tracing.h:365:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^
.output/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
.output/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:18:1: note: expanded from here
___bpf_kprobe_args1
^
.output/bpf/bpf_tracing.h:355:33: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                                       ^
biosnoop.bpf.c:65:5: warning: cast to 'void *' from smaller integer type 'int' [-Wint-to-void-pointer-cast]
.output/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^
.output/bpf/bpf_tracing.h:365:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^
.output/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
.output/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:18:1: note: expanded from here
___bpf_kprobe_args1
^
.output/bpf/bpf_tracing.h:355:25: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                               ^
2 warnings generated.
builder@powerpc64le:~/RPM/BUILD/bcc-0.18.0/libbpf-tools$ /usr/sbin/bpftool gen skeleton .output/biosnoop.bpf.o > .output/biosnoop.skel.h
libbpf: failed to find BTF for extern 'PT_REGS_PARM1': -2
Error: failed to open BPF object file: No such file or directory

[vt-alt]

Defining instead of __TARGET_ARCH_ppc64le __TARGET_ARCH_powerpc causes different error:

builder@powerpc64le:~/RPM/BUILD/bcc-0.18.0/libbpf-tools$ clang -g -O2 -target bpf -D__TARGET_ARCH_powerpc -I.output -c biosnoop.bpf.c -o .output/biosnoop.bpf.o
biosnoop.bpf.c:65:5: error: no member named 'gpr' in 'struct pt_regs'
int BPF_KPROBE(blk_account_io_merge_bio, struct request *rq)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.output/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^~~~~~~~~~~~~~~~~~~~~~~~
.output/bpf/bpf_tracing.h:365:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.output/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
<scratch space>:18:1: note: expanded from here
___bpf_kprobe_args1
^
.output/bpf/bpf_tracing.h:355:33: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                                       ^~~~~~~~~~~~~~~~~~
.output/bpf/bpf_tracing.h:235:32: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) ((x)->gpr[3])
                          ~~~  ^
1 error generated.

[vt-alt]

It seems that vmlinux.h does not have struct pt_regs defined for powerpc.

[anakryiko]

So for ppc64le libbpf doesn't even define PT_REGS_xxx macros. If you are familiar with calling conventions, feel free to contribute definitions (you'll have to send patches to bpf@vger.kernel.org, see libbpf's README).

For powerpc, if it doesn't specify struct pt_regs, what equivalent does it specify?

[anakryiko]

Ok, so it seems for powerpc it's similar situation to s390x, where it has struct user_pt_regs. Can you confirm that your powerpc vmlinux.h has user_pt_regs? We can do the same thing as bpf_tracing.h does for s390.

[vt-alt]

vmlinux.h on powerpc defines this:

root@powerpc64le:~/RPM/BUILD/kernel-image-un-def-5.10.13-alt2# bpftool btf dump file /sys/kernel/btf/vmlinux format c | less
...
struct user_pt_regs {
        long unsigned int gpr[32];
        long unsigned int nip;
        long unsigned int msr;
        long unsigned int orig_gpr3;
        long unsigned int ctr;
        long unsigned int link;
        long unsigned int xer;
        long unsigned int ccr;
        long unsigned int softe;
        long unsigned int trap;
        long unsigned int dar;
        long unsigned int dsisr;
        long unsigned int result;
};

struct pt_regs {
        union {
                struct user_pt_regs user_regs;
                struct {
                        long unsigned int gpr[32];
                        long unsigned int nip;
                        long unsigned int msr;
                        long unsigned int orig_gpr3;
                        long unsigned int ctr;
                        long unsigned int link;
                        long unsigned int xer;
                        long unsigned int ccr;
                        long unsigned int softe;
                        long unsigned int trap;
                        long unsigned int dar;
                        long unsigned int dsisr;
                        long unsigned int result;
                };
        };
        union {
                struct {
                        long unsigned int ppr;
                        long unsigned int kuap;
                };
                long unsigned int __pad[2];
        };
};
...

But, libbpf-tools/vmlinux.h seems to be x86_64 specific. Perhaps, you should put vmlinux.h for all supported architectures, or make libbpf-tools compile with local kernel headers (not pre-packaged vmlinux.h).

[vt-alt]

So for ppc64le libbpf doesn't even define PT_REGS_xxx macros. If you are familiar with calling conventions, feel free to contribute definitions (you'll have to send patches to bpf@vger.kernel.org, see libbpf's README).

Perhaps, this libbpf-tools/Makefile definition could be extended:

ARCH := $(shell uname -m | sed 's/x86_64/x86/')

With s/ppc64le/powerpc/.

[anakryiko]

But, libbpf-tools/vmlinux.h seems to be x86_64 specific. Perhaps, you should put vmlinux.h for all supported architectures, or make libbpf-tools compile with local kernel headers (not pre-packaged vmlinux.h).

Yeah, it is. I somehow missed that. It would be good to have per-architecture vmlinux.h. We can modify Makefile to pick the right one based on the architecture. It would be great if you could generate vmlinux.h with similar config I used for vmlinux.h for these architecture you care about.

Perhaps, this libbpf-tools/Makefile definition could be extended:

ARCH := $(shell uname -m | sed 's/x86_64/x86/')

With s/ppc64le/powerpc/.

Is powerpc and ppc64le the same thing? If yes, then sure, why not?

[vt-alt]

Is powerpc and ppc64le the same thing?

Seems so. ppc64le is what arch outputs, powerpc is Linux' architecture name. The same is with aarch64 and arm64.

[vt-alt]

It would be great if you could generate vmlinux.h with similar config I used for vmlinux.h for these architecture you care about.

That would require kernel source, though. Or booted kernel. Both ways aren't too handy for users, or packagers.

[anakryiko]

Is powerpc and ppc64le the same thing?

Seems so. ppc64le is what arch outputs, powerpc is Linux' architecture name. The same is with aarch64 and arm64.

Ok, cool, then we can add translations for both of them.

It would be great if you could generate vmlinux.h with similar config I used for vmlinux.h for these architecture you care about.

That would require kernel source, though. Or booted kernel. Both ways aren't too handy for users, or packagers.

No, I meant you can build kernel for desired architectures locally, run `bpftool btf dump file > /vmlinux.h, and we'll check in those vmlinux.h files, so that they are readily available in libbpf-tools.

[vt-alt]

No, I meant you can build kernel for desired architectures locally, run `bpftool btf dump file > /vmlinux.h, and we'll check in those vmlinux.h files, so that they are readily available in libbpf-tools.

For example I want to distribute appropriate vmlinux.h together with bpftool package, can you suggest it's location in the system or includes directories?

In that case I may try to create vmlinux-h package for all our architectures (with just vmlinux.h) for latest kernel release and your BPF-preferable config. You can then include vmlinux.h from it into libbpf-tools. We have such architectures: aarch64, armh (32-bit arm), i586, ppc64le, x86_64.

[anakryiko]

Why not just check them in here? We already have vmlinux.h, we can move it to x86/vmlinux.h and have corresponding powerpc/vmlinux.h, etc. Makefile would add proper -I to make #include "vmlinux.h" work?

vmlinux.h shouldn't be distributed with bpftool, it has nothing to do with it.

[vt-alt]

Yes, not with bpftool, I meant together with libbpf-devel package. Related — because it's a requirement to compile for libbpf CO-RE.

I can post vmlinux.h here but isn't it will be obsoleted quickly? With the next version of Linux.

[anakryiko]

It won't get obsoleted very quickly, current vmlinux.h hasn't been updated frequently and is working just fine. vmlinux.h is not strictly required for BPF CO-RE. You can declare types and fields you need from kernel manually, like so:

struct task_struct {
    int pid;
    int tgid;
} __attribute__((preserve_access_index));

And this would work fine. So vmlinux.h is a huge convenience, but is not a mandatory part of CO-RE.

As for including vmlinux.h with any package (bpftool or libbpf, doesn't matter). This is not going to work, because specific generated vmlinux.h depends on exact kernel configuration that was used to compile vmlinux image, from which vmlinux.h was generated. So there is no "golden" version that can be distributed. vmlinux.h in libbpf-tools is built from default x86_64 kernel with few extra BPF-related configs turned on. See https://github.com/iovisor/bcc/pull/2820 for how I built it for x86_64. Let's do the same for other architectures.

[vt-alt]

This is vmlinux.h from Linux 5.10.13 on powerpc/ppc64le (which we build for our repo) with all your BPF options from #2820 applied: vmlinux.h.gz Also, I attach the resulting config.gz, since this is different from pure defconfig (I was unable to boot kernel with pure defconfig quickly).

[vt-alt]

There is the same for aarch64/arm64 architecture: vmlinux.h.gz config.gz

[vt-alt]

@anakryiko Is it what you wanted? ps. About our kernel being not pure defconfig — it will only be extended to more enabled options which, I think, is not a bad thing.

[anakryiko]

@vt-alt, thanks, that's what's needed, I hope. Please give https://github.com/iovisor/bcc/pull/3265 a go and see if it works on your architectures. Thanks!

[vt-alt]

JFYI: In the meantime, I am experimentally building libbpf-tools package for ALT for x86_64 only. They will be installed into /usr/bin/ with bpf- prefix for each tool.

[anakryiko]

It would be great if you could coordinate with @olsajiri on location and naming (see https://github.com/iovisor/bcc/pull/3263#issuecomment-774301083).

[vt-alt]

JFYI again. We recently added CONFIG_DEBUG_INFO_BTF=y for our unstable branch of ALT Linux for all architectures (except aarch64) for the LTS kernel (since v5.4.98).

[anakryiko]

@vt-alt, would you mind submitting a PR to update https://github.com/libbpf/libbpf#bpf-co-re-compile-once--run-everywhere to mention these details? Thanks!