cross-compiling for alpine-aarch64

[martinetd]

Just leaving notes here as I don't think it's worth fixing per se, but this was rather a mess, so figured I might as well write it down for myself if nothing else. Feel free to close immediately unless you see something worth discussing.

• normal stuff, exporting a bunch of env vars: LD=aarch64-alpine-linux-musl-ld CC=aarch64-alpine-linux-musl-gcc ARCH=arm64 CROSS_COMPILE=aarch64-alpine-linux-musl- Normally LD and CC should come from CROSS_COMPILE, but $(CROSS_COMPILE)cc does not exist so CC has to be set to make it use gcc instead (could also be worked around with a symlink), and LD is defined without CROSS_COMPILE in the makefile
• alpine-specific: musl doesn't include argparse and one needs to add -largp to the link commands in the makefile.
• cargo stuff:
  • adding --target=aarch64-unknown-linux-musl to the cargo build command goes some way if the target has been added with rustup target add aarch64-unknown-linux-musl previously (couldn't figure how to do that with the system rust and has to use rustup)
  • final link requires setting CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-alpine-linux-musl-gcc as well on top, because why use standard environment variables...
  • the binary addr2line is at a slightly different location, ../sidecar/target/aarch64-unknown-linux-musl/$(if $(DEBUG),debug,release)/addr2line (extra dir after target)
• the final ld relies on path so _binary___sidecar_target_release_addr2line needs to be updated to __binary_sidecar_start=_binary____sidecar_target_aarch64_unknown_linux_musl_release_addr2line

Here's a diff of the makefile for recap, with the env vars above that should contain everything for aarch64:

diff --git a/src/Makefile b/src/Makefile
index 8090f42..89e115c 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -12,7 +12,7 @@ BPFTOOL ?= $(BPFTOOL_OUTPUT)/bootstrap/bpftool

 LIBBPF_SRC := $(abspath ../libbpf/src)
 LIBBPF_OBJ := $(abspath $(OUTPUT)/libbpf.a)
-SIDECAR := ../sidecar/target/$(if $(DEBUG),debug,release)/addr2line
+SIDECAR := ../sidecar/target/aarch64-unknown-linux-musl/$(if $(DEBUG),debug,release)/addr2line
 # Use our own libbpf API headers and Linux UAPI headers distributed with
 # libbpf to avoid dependency on system-wide headers, which could be missing or
 # outdated
@@ -67,7 +67,7 @@ all: retsnoop simfail
 clean:
    $(call msg,CLEAN)
    $(Q)rm -rf $(OUTPUT) retsnoop simfail bpftool
-   $(Q)$(CARGO) clean --manifest-path=../sidecar/Cargo.toml
+   $(Q)$(CARGO) clean --manifest-path=../sidecar/Cargo.toml --target=aarch64-unknown-linux-musl

 .PHONY: cscope
 cscope:
@@ -128,16 +128,16 @@ $(OUTPUT)/mass_attacher.o: $(OUTPUT)/retsnoop.skel.h $(OUTPUT)/calib_feat.skel.h

 $(SIDECAR)::
    $(call msg,CARGO,addr2line)
-   $(Q)$(CARGO) build --manifest-path=../sidecar/Cargo.toml $(if $(DEBUG),,--release)
+   CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-alpine-linux-musl-gcc $(Q)$(CARGO) build --manifest-path=../sidecar/Cargo.toml $(if $(DEBUG),,--release) --target=aarch64-unknown-linux-musl

 $(OUTPUT)/addr2line.embed.o: $(SIDECAR)
    $(call msg,LLVM_STRIP,$<)
    $(Q)$(LLVM_STRIP) -g $<
    $(call msg,LD,$@)
    $(Q)$(LD) -r -b binary \
-       --defsym __binary_sidecar_start=_binary____sidecar_target_$(if $(DEBUG),debug,release)_addr2line_start \
-       --defsym __binary_sidecar_end=_binary____sidecar_target_$(if $(DEBUG),debug,release)_addr2line_end \
-       --defsym __binary_sidecar_size=_binary____sidecar_target_$(if $(DEBUG),debug,release)_addr2line_size \
+       --defsym __binary_sidecar_start=_binary____sidecar_target_aarch64_unknown_linux_musl_$(if $(DEBUG),debug,release)_addr2line_start \
+       --defsym __binary_sidecar_end=_binary____sidecar_target_aarch64_unknown_linux_musl_$(if $(DEBUG),debug,release)_addr2line_end \
+       --defsym __binary_sidecar_size=_binary____sidecar_target_aarch64_unknown_linux_musl_$(if $(DEBUG),debug,release)_addr2line_size \
        -o $@ $<

 $(OUTPUT)/addr2line.o: $(OUTPUT)/addr2line.embed.o
@@ -157,7 +157,7 @@ retsnoop: $(addprefix $(OUTPUT)/,                   \
              mass_attacher.o)                  \
      $(LIBBPF_OBJ)
    $(call msg,BINARY,$@)
-   $(Q)$(CC) $(CFLAGS) $^ -lelf -lz -o $@
+   $(Q)$(CC) $(CFLAGS) $^ -lelf -lz -largp -o $@

 $(OUTPUT)/tests/simfail.o: $(OUTPUT)/tests/kprobe_bad_kfunc.skel.h \
               $(OUTPUT)/tests/fentry_unsupp_func.skel.h    \
@@ -168,7 +168,7 @@ simfail: $(addprefix $(OUTPUT)/tests/,                  \
             simfail.o)                     \
      $(LIBBPF_OBJ)
    $(call msg,BINARY,$@)
-   $(Q)$(CC) $(CFLAGS) $^ -lelf -lz -o $@
+   $(Q)$(CC) $(CFLAGS) $^ -lelf -lz -largp -o $@

 # delete failed targets
 .DELETE_ON_ERROR:

As a side note, if instead of cross compiling one goes the qemu-user route, cargo has a bug that makes it consume insane amount of memory when fetching its crate index while running under qemu-user. That can be worked around by exporting CARGO_NET_GIT_FETCH_WITH_CLI=true before running make, and this worked for me -- I'll be using the binary generated that way for now.

[anakryiko]

@martinetd I'm open to Makefile changes to make this easier, retsnoop's Makefile is not as crazy as, say, selftests/bpf/Makefile in Linux repo, so adding support for CROSS_COMPILE would be fine with me. If the musl stuff can be more or less isolated to one or two checks, that should be fine as well.

As for _binary____sidecar_target_release_addr2line_start not being there. This symbol name is automatically derived from the path (you changed it to ../sidecar/target/aarch64-unknown-linux-musl/release/addr2line, so it will have extra _aarch64_unknown_linux_musl_ in the middle, I suspect. This is objcopy's convention, I don't have much control over this. But again, all that probably can be accommodated.

So if you feel like this has to be addressed, please send PR and let's talk about it.

[martinetd]

Ah, I suspected something with the path for that ld symbol, but I kept the dashes in the target triplet instead of underscores, and ld didn't like dashes. Well, that sure works better and I've updated the first post with full instructions and an ugly diff for reference/recap, but nothing that could be merged as is.

As said initially I don't think it's necessarily worth fixing at this point -- unless someone comes in who want to e.g. package it for yocto, I'm not aware of any distribution doing cross build, I mostly looked because cargo + qemu-user was being stupid. In particular I'm not sure how we could possibly do the mappings for cargo target and its env var; guessing the libc from CROSS_COMPILE doesn't sound like a good idea, so it's probably best to just leave some instructions instead? I don't know. If someone looks they might be able to find this issue ;)

Anyway, if it turns out I use retsnoop a bit I might look into packaging it for alpine, in which case I might send a patch just for argp, but it's a bit early for me at this point as it will be the first time I use it on an actual problem so I don't want to commit to anything right now.

[anakryiko]

@martinetd can this be closed?

[martinetd]

I'd say cross compiling still isn't quite easy, but building the sidecar separately and pasing -largp through LDFLAGS ought to work so it should have gotten quite a bit easier than last time I tried -- and I can't think of anything that'd be easy to improve anyway.

Thanks for the reminder, let's close this for now!