Document fscrypt cross-compiling process

[M1cha]

Apparently cross-compiling fscrypt doesn't work:

$ env GOOS=linux GOARCH=arm make
go build --ldflags '-s -w -X "main.version=v0.2.3-30-g0f451a7" -X "main.buildTime=Sat Sep  1 23:10:04 CEST 2018" -extldflags ""' -o bin/fscrypt ./cmd/fscrypt
# github.com/google/fscrypt/security
security/keyring.go:157:22: undefined: getUids
security/keyring.go:159:12: undefined: setUids
security/keyring.go:163:16: undefined: setUids
# github.com/google/fscrypt/filesystem
filesystem/filesystem.go:359:15: undefined: makeLink
filesystem/filesystem.go:399:15: undefined: getMountsFromLink
make: *** [Makefile:88: bin/fscrypt] Error 2

This problem looks really weird since all the symbols it's complaining about are actually there.

[josephlr]

@M1cha Cross compiling with CGO (which fscrypt uses) can be a bit of a pain, so here are the basic steps.

• You need to get all the components to build fscrypt, but for the target architecture. This means you need:

  • A GCC cross compiling toolchain from your host to the target architecture.
  • A version of libpam (and its headers) for the target architecture.
  • Using the go toolchain with GOARCH set the the target architecture.

• Once you have everything installed, a cross compile build can be invoked like:

  GOARCH=arm CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc make

• This gives you arm output files:

  > file bin/fscrypt
  bin/fscrypt: ELF 32-bit LSB executable, ARM, EABI5 ...
  > file bin/pam_fscrypt.so
  bin/pam_fscrypt.so: ELF 32-bit LSB shared object, ARM, EABI5

[josephlr]

I changed this to a documentation issue, mainly because the full process is really hairy.

Here are the full steps to get the above commands working on my Ubuntu Bionic 18.04 system:

• Update /etc/apt/sources.list to properly download the correct architectures.
  • For Debian based systems, see this guide.
  • For example, my /etc/apt/sources.list looks like this.
  • sudo apt update && sudo apt upgrade
• Install libpam0g-dev for your target architecture:
  • sudo apt install libpam0g-dev:armhf
• Install a gcc cross-compiling toolchain for your target architecture:
  • sudo apt install gcc-arm-linux-gnueabihf
• Invoke the Go toolchain with the proper environment variables:
  • GOARCH=arm
  • CGO_ENABLED=1 (as cross compiling disables CGO by default)
  • CC=arm-linux-gnueabihf-gcc
  • Can invoke with make
  • GOARCH=arm CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc make
  • Can invoke the go command directly
  • GOARCH=arm CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc go build ./cmd/fscrypt

This can also be confusing when Debian, GNU, and Go all use different names to refer to architectures. The following chart is helpful (source):

Go's GOOS	Debian's arch	GNU's target triple
386	i386	i686-linux-gnu
amd64	amd64	x86_64-linux-gnu
arm64	arm64	aarch64-linux-gnu
arm	armhf	arm-linux-gnueabihf
ppc64le	ppc64el	powerpc64le-linux-gnu
s390x	s390x	s390x-linux-gnu

EDIT: Of course there are other GOOS values not listed here, but they are not supported by Ubuntu, so they are probably less relevant for cross compiling.

[BlueMax]

Any idea why this happens? Toolchain is musl-cross. Host OS is Arch Linux. Edit: Also tried: GO_FLAGS=-buildmode=pie GOARCH=arm64 CGO_ENABLED=1 make [...] This only happens on '-static' build. Shared does work. I think this is because libpam is compiled as 'shared' always.

$ GOARCH=arm64 CGO_ENABLED=1 make LDFLAGS="-static -ldl -laudit -lcap-ng" CFLAGS="-O2 -g0 -s -pipe -pthread -Wp,-I/home/user/android/sysroot-aarch64/usr/include -Wl,-L/home/user/android/sysroot-aarch64/usr/lib" PREFIX="/tmp/myout" CGO_CFLAGS="--sysroot /home/user/android/sysroot-aarch64" CGO_LDFLAGS="--sysroot /home/user/android/sysroot-aarch64"
go build --ldflags '-s -w -X "main.version=v0.2.7-44-g636698b" -extldflags "-static -ldl -laudit -lcap-ng"' -trimpath -o bin/fscrypt ./cmd/fscrypt
# github.com/google/fscrypt/cmd/fscrypt
/usr/lib/go/pkg/tool/linux_amd64/link: running /home/user/android/gcc-musl-aarch64/bin/aarch64-linux-musl-gcc failed: exit status 1
/media/LinuxPart2/musl-cross-make/output64/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: read-only segment has dynamic relocations
collect2: error: ld returned 1 exit status

make: *** [Makefile:95: bin/fscrypt] Fehler 2

Edit: The solution is -buildmode=exe. Not tested the binary yet but at least it does compile.

GO_FLAGS=-buildmode=exe GOARCH=arm64 CGO_ENABLED=1 make LDFLAGS="-static -ldl -laudit -lcap-ng" CFLAGS="-O2 -g0 -s -static -pipe -pthread -Wp,-I/home/user/android/sysroot-aarch64/usr/include -Wl,-L/home/user/android/sysroot-aarch64/usr/lib" PREFIX="/tmp/myout" CGO_CFLAGS="--sysroot /home/user/android/sysroot-aarch64" CGO_LDFLAGS="--sysroot /home/user/android/sysroot-aarch64"

[JasonShigit]

@josephlr Could you provide more details regarding "how to cross-compile with rootfs" ? When I build with "GOARCH=arm64 CGO_ENABLED=1 CC=aarch64-linux-gnu-gcc make" then i get: /usr/lib/gcc-cross/aarch64-linux-gnu/7/../../../../aarch64-linux-gnu/bin/ld: cannot find -lpam /usr/lib/gcc-cross/aarch64-linux-gnu/7/../../../../aarch64-linux-gnu/bin/ld: cannot find -lpam

and i try to fix it with below parammeter(libpam is already installed in arm-rootfs): CGO_CFLAGS="--sysroot /home/work/arm-rootfs" -lpam is also cannot be found.

[josephlr]

I'm looking some more into this and would like some feedback on use-cases.

It looks like both cases are 64bit arm, on these systems, would you need to use user login passwords (i.e. do you need to make sure the encryption password matches the user's login passphrase), or do you just need normal passwords?

@JasonShigit which Linux distro are you using for the host? I know @BlueMax was using Arch Linux.

If it turns out user passphrases are not needed, we can just add a flag to disable the PAM dependency at compile time. This should remove all need for cgo and libpam, which should make cross compiling much, much easier.

[josephlr]

@JasonShigit it looks like the linker can't find libpam because the search path is incorrect. Fixing this might be distro/toolchain specific. I'm not familiar with using a rootfs for this case. Do you have more information?

[JasonShigit]

I'm looking some more into this and would like some feedback on use-cases.

It looks like both cases are 64bit arm, on these systems, would you need to use user login passwords (i.e. do you need to make sure the encryption password matches the user's login passphrase), or do you just need normal passwords?

@JasonShigit which Linux distro are you using for the host? I know @BlueMax was using Arch Linux.

If it turns out user passphrases are not needed, we can just add a flag to disable the PAM dependency at compile time. This should remove all need for cgo and libpam, which should make cross compiling much, much easier.

Thanks for your reply. My host OS is ubuntu18.04, and arm target os is ubuntu18.04 too. So user passphrases is required, i think. Issue fixed with CGO_LDFLAGS="--sysroot /home/work/arm-rootfs" GOOS=linux GOARCH=arm64 CGO_ENABLED=1 CC=aarch64-linux-gnu-gcc CGO_LDFLAGS="--sysroot /home/work/arm-rootfs" make

[directionless]

In case it helps folks, I've been able to cross compile from macOS to linux, using zig. Basically following the instructions at https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho (with a little adjustment to translate GOOS and GOARCH

[VenkateshKaduru]

can some explain the below: $make mkdir -p /home/venk/mxcam/P7/mxcam74011/pkg_root/target/lib ( cd fscrypt-0.2.5 && \ PKG_CONFIG_PATH=/home/venk/mxcam/P7/mxcam74011/pkg_root/target/usr/lib/pkgconfig PKG_ROOT=/home/venk/mxcam/P7/mxcam74011/pkg_root/target \

--host=arm-v7a-linux-gnueabihf --build=i686-pc-linux-gnu \

#aarch64-linux-gnu \
#--without-selinux  --without-libmount --without-blkid --enable-dependency-tracking  \
CFLAGS="-mcpu=cortex-a53 -mfpu=crypto-neon-fp-armv8   -flto=16  -Wundef  -DGLIBC_VERSION=022800 -DHAS_WINPACKAGE=0 -DHOST_TOOL=0 -DOPENSSL_MINIMAL=0 -DKERNEL_MODULE_SET=3 -DHAS_DEVICETREE=1 -DHAS_EMMC=1 -DMIC_BLOW_FUSE=0 -DVOIP_VERSION=3 -DWITH_H264=1 -DANDROMEDA=1 -DWITH_ALSA=0 -DWITH_ONVIF=1 -DWITH_STRATOCAST=1 -DAUDIOTEST_LEVELS=320 -DMX_PLUGINS=1 -DMX_PLUGINS_RAW_IMAGE=1 -DWITH_LXC=0 -DARCH_ARM -DBAYER_SUPPORT=0 -DP7_FRAMEHEADER=1 -DBOOST_PP_VARIADICS_MSVC=0 -D_FILE_OFFSET_BITS=64 -DAS2V1_INTEGRATED=1 -DMXACTIVITY_SENSOR_USE_DESIRED_FRAME_RATE=1 -DIMA_DENOISING=1 -DTHERMAL_EXTREMUM_POSITION=1 -DWITH_PYTHON=0 -DMX_OEM=0 -DNUMOFACTIONHANDLERS=20 -DMXANALYTICS_INTEGRATED=1 -DMXANALYTICS_PEOPLE_COUNTING=1 -DMXANALYTICS_HEAT_MAP=1 -std=gnu99  -O2 -w -Wno-undef -std=gnu11  -I/home/venk/mxcam/P7/mxcam74011/pkg_root/target/usr/include/security" \
LDFLAGS="-mcpu=cortex-a53 -mfpu=crypto-neon-fp-armv8   -flto=16   -O2 -L/home/venk/mxcam/P7/mxcam74011/pkg_root/target/usr/lib -L/home/venk/mxcam/P7/mxcam74011/pkg_root/target/lib" \
gccgoflags=" -L/home/venk/mxcam/P7/mxcam74011/pkg_root/target/usr/local/go/bin" \
#CGO_LDFLAGS=" -L/home/venk/mxcam/P7/mxcam74011/pkg_root/target/usr/arm-linux-gnueabihf/lib " \
GOARCH=arm \
CGO_ENABLED=1 \
CC=gcc-arm-linux-gnueabihf \
make \

) make[1]: Entering directory '/home/venk/mxcam/P7/mxcam74011/source-base/fscrypt/fscrypt-0.2.5' go build --ldflags '-s -w -X "main.version=MX-V5-7-0-24-39-g0160143edf" -X "main.buildTime=Mo 9. Mai 16:57:11 CEST 2022" -extldflags ""' -o bin/fscrypt ./cmd/fscrypt go build: when using gccgo toolchain, please pass linker flags using -gccgoflags, not -ldflags

github.com/urfave/cli

x86_64-linux-gnu-gccgo-9: error: unrecognized command line option ‘-marm’; did you mean ‘-mabm’?

github.com/pkg/errors

x86_64-linux-gnu-gccgo-9: error: unrecognized command line option ‘-marm’; did you mean ‘-mabm’?

golang.org/x/crypto/blake2b

x86_64-linux-gnu-gccgo-9: error: unrecognized command line option ‘-marm’; did you mean ‘-mabm’?

github.com/golang/protobuf/proto

x86_64-linux-gnu-gccgo-9: error: unrecognized command line option ‘-marm’; did you mean ‘-mabm’?

golang.org/x/crypto/hkdf

x86_64-linux-gnu-gccgo-9: error: unrecognized command line option ‘-marm’; did you mean ‘-mabm’?

golang.org/x/sys/unix

x86_64-linux-gnu-gccgo-9: error: unrecognized command line option ‘-marm’; did you mean ‘-mabm’? make[1]: [Makefile:89: bin/fscrypt] Error 2 make[1]: Leaving directory '/home/venk/mxcam/P7/mxcam74011/source-base/fscrypt/fscrypt-0.2.5' make: [Makefile:34: done/configure] Error 2

[VenkateshKaduru]

how to build for 32bit?

• arm-v7a-linux-gnueabihf-gcc or is there any way to add -m64? as command arg.

github.com/google/fscrypt/security

arm-v7a-linux-gnueabihf-gcc: error: unrecognized command line option '-m64'

github.com/google/fscrypt/filesystem

arm-v7a-linux-gnueabihf-gcc: error: unrecognized command line option '-m64'

github.com/google/fscrypt/security

arm-v7a-linux-gnueabihf-gcc: error: unrecognized command line option '-m64'

github.com/google/fscrypt/filesystem

arm-v7a-linux-gnueabihf-gcc: error: unrecognized command line option '-m64' make[1]: [Makefile:89: bin/fscrypt] Error 2 make[1]: Waiting for unfinished jobs.... make[1]: [Makefile:92: bin/pam_fscrypt.so] Error 2 make[1]: Leaving directory '/home/venk/mxcam/P7/mxcam74011/source-base/fscrypt/fscrypt-0.2.5' make: [Makefile:91: done/build] Error 2

[blubbafett]

Anyone who has managed to build a static build of this for arm or aarch64? Dynamically linked works fine, but static....