Add support for writing maps into .maps ELF section in aya-bpf

[v-thakkar]

At the moment generating BTF debug info by default is not enabled. Once LLVM fixes are in and this is merged in bpf-linker, we'll need to teach aya-bpf to support writing maps into .maps section.

Adding this as an issue, so that I don't forget to tackle it once the above 2 things are done. cc @vadorovsky

[vadorovsky]

We have some updates and a bit of progress here.

Discord thread for anyone interested

LLVM fix

@dave-tucker submitted this LLVM fix https://reviews.llvm.org/D134533 - the only thing missing to get it merged is writing some LLVM IR test for that, which Dave is working on.

Building LLVM with this patch and then building bpf-linker with it makes it possible to build eBPF programs with debug_info and BTF. :tada:

Actual BTF map support in Aya

I crated this repo where I'm trying to make a small aya-bpf program with a BTF map:

https://github.com/aya-rs/aya-btf-maps https://github.com/aya-rs/aya-btf-maps/blob/main/fork-ebpf/src/main.rs

It's using libbpf 1.0 in the userspace and Aya in the eBPF crate.

It's not fully working yet, libbpf complains about the type field:

libbpf: map 'PID_MAP': attr 'type': expected PTR, got int.

It most likely means that there is something behind __type and __u32 macros here which I missed. UPDATE (5.10.2022): this is solved now

These are examples of BTF map definitions preprocessed by clang:

struct {
 int (*type)[BPF_MAP_TYPE_ARRAY];
 typeof(u32) *key;
 typeof(u32) *value;
 int (*max_entries)[256];
} array1

struct {
 int (*type)[BPF_MAP_TYPE_SK_STORAGE];
 int (*map_flags)[BPF_F_NO_PREALLOC];
 typeof(int) *key;
 typeof(int) *value;
} sk_stg_map

I managed to build such a struct with this macro:

https://github.com/vadorovsky/aya-btf-maps/blob/0f952634bf962e78bca54b6f19f9baacf0c868ae/aya-btf-macros/src/expand.rs#L55-L73

libbpf is fine with how the .maps section looks like, but the kernel doesn't like the BTF included in our program:

Error: failed to load bpf object: -22

We need to compare .debug_info produced by clang and Rust and hunt the differences. We will probably need to solve them with some hacks in bpf-linker first, then come up with some proper fixes for LLVM or rustc or kernel, wherever the problem happens.

Loading the same program with Aya (in userspace) works.

Please note that this example is just defining a map as a struct directly in the eBPF program crate. And we use bindings to BPF helpers directly. Once we get it working, we still need to figure out how we design a proper API for BTF maps in Aya.

If anyone wants to try our or/and help, I added instructions to README how to prepare patched LLVM, bpf-linker, build the project and what's done there.

[vadorovsky]

Currently we are doing a following fixup in Aya (userspace, when loading programs):

https://github.com/aya-rs/aya/blob/66b4f79ecafe9832fcc1e44373598774b9954514/aya/src/obj/btf/btf.rs#L393-L401

PTR looks like that in BTF (produced by clang):

[1] PTR '(anon)' type_id=3
[...]
[5] PTR '(anon)' type_id=6
[...]
[7] PTR '(anon)' type_id=8
[...]
[11] PTR '(anon)' type_id=0

PTRs in Rust BTF have names corresponding to the type of a pointer:

[1] PTR '*const [i32; 1]' type_id=3
[...]
[5] PTR '*const u32' type_id=6
[...]
[7] PTR '*const [i32; 1024]' type_id=8
[...]
[9] PTR '*const [i32; 0]' type_id=10
[...]
[13] PTR '*mut core::ffi::c_void' type_id=14
[...]
[17] PTR '*mut u8' type_id=18

DWARF (produced by clang):

0x000000ca:   DW_TAG_pointer_type
                DW_AT_type      (0x000000cf "int[1]")
[...]
0x000000df:   DW_TAG_pointer_type
                DW_AT_type      (0x000000e4 "unsigned int")
[...]
0x00000102:   DW_TAG_pointer_type
                DW_AT_type      (0x00000107 "long (const char *, __u32, ...)")

(notice the lack of DW_AT_name field)

We need to move it to bpf-linker. And all the other differences in debug_info between eBPF objects commpiled with C and Rust - also have to be handled in bpf-linker. At least for now (we might consider proper fixes in rustc/LLVM/kernel when we know everything and get stuff working).

Probably the best place for fixup would be somewhere around here:

https://github.com/aya-rs/bpf-linker/blob/0666b6424216ad27651f14b0f433f6200872aff8/src/llvm/mod.rs#L213-L220

On this stage, we have already one big LLVM module after linking, we just need to find a way to find the debug info of that module and be able to iterate over it.

Quotes from Discord:

you probably need to create an iterator (see iter.rs) based on this https://docs.rs/llvm-sys/latest/llvm_sys/core/fn.LLVMGetFirstNamedMetadata.html filter all the debug metadata recursively visit all the metadata build some kind of id => metadata mapping have some kind of sentinel for metadata id that has been referenced but you haven't found yet then after you finish scanning, scan the whole id => metadata mapping look for sentinels that still exists (things for which you have missing definitions) and create stubs or you can probably even generate the stubs as you go and then replace them if you encounter their actual definition or I mean, I don't remember what that particular crash was about maybe we can get away by generating stubs just for that the long term solution is what I'm doing with the #[btf_export] thing (which now that I looked at the code is easier than I thought)

There was also an idea of modifying DI by writing an LLVM pass, but

I don't think we need a custom pass? there's already "passes" in bpf-linker that don't use the actual llvm pass interface you need to implement that interface if you want it to be called by the pass manage but we don't want that https://github.com/aya-rs/bpf-linker/blob/main/src/llvm/mod.rs#L21 internalize is something we do to massage linkage essentially, bpf-linker gets all the input modules, merges in one big module, that big module contains everything you need including DI the fix would consist in visiting all the DI, and fixing missing definitions

[davibe]

I need this for a project I'm working on, so I've started implementing doing the fixups in bpf-linker