Doubt about design decision

[rafaeldtinoco]

Hello, Id like to clarify your design decisions (for learning purposes and to share knowledge, hopefully, both ways), if you allow me. Sorry to post you an issue, but you didn't have "discussions" option set.

Based on your documentation, you say:

Instrumentation Stability
eBPF programs access user code and variables by analyzing the stack and the CPU registers. For example, to read the value of the target field in the google.golang.org/grpc.ClientConn struct (see gRPC instrumentor for an example), the eBPF program needs to know the offset of the field inside the struct. The offset is determined by the field location inside the struct definition.

This sounds to me like .BTF and .BTF.ext sections in eBPF objects, placed by LLVM, so libbpf can read the RELO information and calculate relocations based on some speculations.

Hard coding this offset information into the eBPF programs creates a very unstable instrumentation. Fields locations inside structs are subject to change and the eBPF program needs to be recompiled every time the struct definition changes.

Wouldn't it be possible to make golang compiler to place .BTF/.BTF.ext like information into the generated objects ? Just like LLVM does for eBPF objects into specific ELF sections ?

Luckily for us, there is a way to analyze the target binary and extract the required offsets, by using DWARF. The DWARF debug information is generated by the compiler and is stored inside the binary.

This is similar of creating an external BTF files w/ pahole, representing kernels (which is analog to the userland binary you will place uprobes).

Notice that one of our design goals is to support stripped Go binaries - meaning binaries that do not contain debug information. In order to support stripped binaries and to create a stable instrumentation, we created a library called [offsets-tracker](https://github.com/keyval-dev/offsets-tracker). This library tracks the offset of different fields across versions.

Couldn't you generate RAW BTF files from all existing DWARF symbols ? And then use the BTF info (in a more generic way, instead of only picking up offsets for particular structs, you could get for all types used).

[edeNFed]

Hi @rafaeldtinoco Thanks a lot for the feedback! Let me start by saying this is my first big eBPF project, so you probably know better than I do. I can replace the offsets tracking with BTF format and it sounds like a much better solution. The only question I have is will libbpf be able to relocate structs in userspace? If I understand correctly, relocation works by comparing the BTF info of the program with the BTF info of the currently running kernel. How is it going to work for structs in a Go binary that does not provide any debug information? What is going to be compared against the eBPF program BTF info?

[rafaeldtinoco]

Hi @rafaeldtinoco Thanks a lot for the feedback! Let me start by saying this is my first big eBPF project, so you probably know better than I do.

Nah, we're all here to learn and your project seems very nice =D. Please forgive me if Im spending your time. I'm more thinking out loud here than proposing you changes (thinking on things that could be useful in our case as well).

Item 1

I can replace the offsets tracking with BTF format and it sounds like a much better solution.

This is what I thought.. as an example I have example-static (an elf executable binary from go compiler) and example-c-static (an elf executable binary from c compiler):

Problem here is that pahole is only converting the TYPES (.BTF ELF section) and not generating the information contained in .BTF.ext ELF section (which LLVM creates for BPF architecture) like function_info and line_info (+ type relocation info). Although the function_info is an offset to each ELF section (as in eBPF the sections are different programs and a single function) and not an offset to each ELF symbol (I need to check this).

To be able to set the uprobes you would need more than that (symbol offset). I'm not sure it would work.

Item 2

Within libbpfgo, we also have to read elf and its symbols:

The only question I have is will libbpf be able to relocate structs in userspace?

That is where I was going.. to have "BTF" encoded into a binary, then we would know symbols offsets (if function_info of .BTF.ext provided that) and type info and relocation needs. But I guess there are many "TODOs" so it won't be an option =(.

If I understand correctly, relocation works by comparing the BTF info of the program with the BTF info of the currently running kernel.

Exactly!

How is it going to work for structs in a Go binary that does not provide any debug information?

In my original idea, let's say you had symbol offsets info generated within the external BTF files, you could generate external BTF files to existing binaries from their DWARF info (like you're doing now), and use those external BTF files (per module version) to have symbols offsets.. but it won't work as you don't have that information there.

My 2nd thought was if we had info about different types as well, for different binary versions (and BTF already provides that) then the userland program (receiving data from kernel uprobe) would be able to calculate offsets of diff types, depending on the module/binary version <- this is similar to eBPF CO-RE concept (but inverse maybe).

Anyway, good exercise of thinking... thanks for replying back.

[edeNFed]

Thank you for the detailed explanation it was great! I really liked the idea of replacing the current custom JSON format with BTF. I will surely add it to our roadmap to further explore this direction.

[rafaeldtinoco]

I'll keep following you, then. Nice work! Sorry for the noise. If you ever want to dig into libbpf and how it makes the relocations (https://github.com/aquasecurity/btfhub/blob/main/docs/btfgen-internals.md). Good luck!