[FEATURE] Try to run eBPF with bpftime on MacOS

[yunwei37]

Is your feature request related to a problem? Please describe.

Since bpftime can run entirely in userspace, can we port bpftime to MacOS?

Describe the solution you'd like

Try to run it on MacOS.

[ShawnZhong]

I have attempted to port bpftime to macOS.

1. I am able to compile bpftime itself (without any dependencies) on macOS with some minor tweaks, which is quite nice.

2. bpftime depended on libbpf to manage the bpf object file (e.g., calling bpf_object__open to open the file, relocate address, init maps, etc). Compiling libbpf itself requires mocking a few linux headers on macOS, but it's doable.

3. libbpf relies on libelf to parse the bpf object file stored in ELF format. Currently, the largest obstacle is that libelf is notoriously difficult to compile on macOS, as discussed in a StackOverflow post:

   However, elfutils has massive portability problems. This is one of the 20+ changes needed to make elfutils build

   The libelf package on HomeBrew is way too outdated (updated in 2009) to be usable for libbpf.

For whoever wants to work on this issue, here are some of my thoughts:

• I don't think it's worth the effort trying to compile libelf on macOS due to the complexity.

• It might be worth trying to port libbpf to use other ELF libraries (e.g., ELFIO or llvm-readobj)

• Another possibility is to bypass libbpf and load the bpf program ourselves.

[yunwei37]

@ShawnZhong Thank you very much for your effort!

In fact, I think libbpf is not necessary for running bpftime in userspace. The syscall-server.so use LD_PRELOAD to override the functions in libc, and doesn't require libbpf to load the bpf programs. I think maybe we can bypass libbpf, and still able to maintain the compatibility of runnng eBPF applications.

The libbpf in bpftime runtime is used for:

1. Integrating with kernel eBPF runtimes, open maps, load and exec some eBPF programs into the kernel while keep some eBPF programs in userspace. This is no need if we want to run bpftime complete in userspace on MacOS, and we can use some option to config them not be compiled.
2. In bpftime/runtime/object, this is a depreciated manual eBPF object loader, which is only used for test now. Maybe you can try to build without this target.
3. Also, there might be some type definitions using libbpf and kernel headers. For example, the helpers.

Perhaps we can use some compile flags to help us build a minimal runtime without these feature supports? I think that's the best approach.

For example, add a BPFTIME_BUILD_USERSPACE (Or other name) option, and compile bpftime with only userspace maps and no kernel eBPF access, so we can get rid of libbpf.

[Officeyutong]

Basic roadmap:

Port bpftime to aarch64 macos, and x86 if possible

• [ ] There are some linux specified syscalls used in bpftime. Find out them and discuss how to replace them with other things when porting to darwin. POSIX calls are suppoed to work on darwin
• [x] Some dependencies such libbpf are unable to work on non-linux platforms. We should be able disable features that used libbpf in bpftime. See #175
• [ ] Frida is needed for uprobe. Need frida on aarch64. Some register names might need to be adapted.
• [ ] Many of our unit tests, especially those related to frida uprobe attach manager, are hardcoded with registers on x86. They need to be adapted.
• [ ] Syscall trace are implemented only for x86 SystemV ABI. We should have a further discussion on whether the similar way could be done on aarch64 (on x86, we rewrite instruction syscall to call *%rax and setup trampolines on page 0 to implement userspace syscall trace)

@hp77-creator @viniciusd

[Officeyutong]

Member

@hp77-creator libbpf was already done, what's your next plan?

[hp77-creator]

I can take up the frida related issues, I tried running bpftime on my system but it doesn't work, had discussion with @viniciusd on this and he mentioned it might be frida, so yea can take up that.

[Officeyutong]

I can take up the frida related issues, I tried running bpftime on my system but it doesn't work, had discussion with @viniciusd on this and he mentioned it might be frida, so yea can take up that.

OK, related things will be discussed at https://github.com/eunomia-bpf/bpftime/issues/304