How to cross compile on macOS for a different macOS architecture

[ryandesign]

I am running macOS 12 on an x86_64 Mac. How can I compile chez scheme for arm64 macOS? I have read the BUILDING document and have not gained an understanding of how to do this. The document refers often to <machine type> without explaining whether this refers to the type of the build machine or the type of the machine for which the build is being done. I already have lz4 and zlib installed universal (compiled for both arm64 and x86_64 in the same binary).

What I have tried:

% CC=/usr/bin/clang CPPFLAGS=-I/opt/local/include LDFLAGS=-L/opt/local/lib ./configure --threads --libkernel --machine=tarm64osx LZ4=-llz4 ZLIB=-lz
Configuring for tarm64osx, and will create boot files via pb
% make
mkdir -p bin
/usr/bin/clang -arch arm64 -O2 -DZUO_LIB_PATH='"'"../zuo/lib"'"' -o bin/zuo ./zuo/zuo.c
bin/zuo tarm64osx MAKE="/Applications/Xcode.app/Contents/Developer/usr/bin/make"
/bin/sh: bin/zuo: Bad CPU type in executable
make: *** [build] Error 126

Naturally I can't run an arm64 executable on an x86_64 machine. This happens because, if CFLAGS is empty, this code sets CFLAGS to contain -arch arm64:

https://github.com/cisco/ChezScheme/blob/f1ad314a3809074c6f2b986a5db6a84ad88700c1/configure#L810-L811

and, if CC_FOR_BUILD is empty, this code sets CC_FOR_BUILD="$CC $CFLAGS":

https://github.com/cisco/ChezScheme/blob/f1ad314a3809074c6f2b986a5db6a84ad88700c1/configure#L833-L834

So I try setting CC_FOR_BUILD myself to a value that does not include -arch flags which failed:

% CC=/usr/bin/clang CC_FOR_BUILD=/usr/bin/clang CPPFLAGS=-I/opt/local/include LDFLAGS=-L/opt/local/lib ./configure --threads --libkernel --machine=tarm64osx LZ4=-llz4 ZLIB=-lz
No suitable machine type found in "./boot".

Available machine types:
  pb

See "./BUILDING" for ways of getting boot files.

This happens because, if CC_FOR_BUILD is set, this code sets enableFrompb=no.

https://github.com/cisco/ChezScheme/blob/f1ad314a3809074c6f2b986a5db6a84ad88700c1/configure#L833-L836

Am I now supposed to add --pb or --pbarch to the configure flags? If so, which one? The descriptions of these flags do not provide any guidance for how one should choose between them.

I tried adding --pbarch which failed:

% CC=/usr/bin/clang CC_FOR_BUILD=/usr/bin/clang CPPFLAGS=-I/opt/local/include LDFLAGS=-L/opt/local/lib ./configure --threads --libkernel --machine=tarm64osx --pbarch LZ4=-llz4 ZLIB=-lz
No suitable machine type found in "./boot".

Available machine types:
  pb

See "./BUILDING" for ways of getting boot files.

I tried instead adding --pb which failed:

% CC=/usr/bin/clang CC_FOR_BUILD=/usr/bin/clang CPPFLAGS=-I/opt/local/include LDFLAGS=-L/opt/local/lib ./configure --threads --libkernel --machine=tarm64osx --pb LZ4=-llz4 ZLIB=-lz
Configuring for pb
% make
mkdir -p bin
/usr/bin/clang -DZUO_LIB_PATH='"'"../zuo/lib"'"' -o bin/zuo ./zuo/zuo.c
bin/zuo pb MAKE="/Applications/Xcode.app/Contents/Developer/usr/bin/make"
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/statics.o -c c/statics.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/segment.o -c c/segment.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/alloc.o -c c/alloc.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/symbol.o -c c/symbol.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/intern.o -c c/intern.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/gcwrapper.o -c c/gcwrapper.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/gc-011.o -c c/gc-011.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/gc-par.o -c c/gc-par.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/gc-ocd.o -c c/gc-ocd.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/gc-oce.o -c c/gc-oce.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/number.o -c c/number.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/schsig.o -c c/schsig.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/io.o -c c/io.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/new-io.o -c c/new-io.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/print.o -c c/print.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/fasl.o -c c/fasl.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/vfasl.o -c c/vfasl.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/stats.o -c c/stats.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/foreign.o -c c/foreign.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/prim.o -c c/prim.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/prim5.o -c c/prim5.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/flushcache.o -c c/flushcache.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/schlib.o -c c/schlib.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/thread.o -c c/thread.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/expeditor.o -c c/expeditor.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/scheme.o -c c/scheme.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/compress-io.o -c c/compress-io.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/random.o -c c/random.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/ffi.o -c c/ffi.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/self-exe.o -c c/self-exe.c
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/c/pb.o -c c/pb.c
ar rc pb/boot/pb/libkernel.a pb/c/statics.o pb/c/segment.o pb/c/alloc.o pb/c/symbol.o pb/c/intern.o pb/c/gcwrapper.o pb/c/gc-011.o pb/c/gc-par.o pb/c/gc-ocd.o pb/c/gc-oce.o pb/c/number.o pb/c/schsig.o pb/c/io.o pb/c/new-io.o pb/c/print.o pb/c/fasl.o pb/c/vfasl.o pb/c/stats.o pb/c/foreign.o pb/c/prim.o pb/c/prim5.o pb/c/flushcache.o pb/c/schlib.o pb/c/thread.o pb/c/expeditor.o pb/c/scheme.o pb/c/compress-io.o pb/c/random.o pb/c/ffi.o pb/c/self-exe.o pb/c/pb.o
/usr/bin/clang -I/opt/local/include -DPORTABLE_BYTECODE -Ipb/boot/pb -Ipb/c -Ic/ -arch arm64 -O2 -Wpointer-arith -Wall -Wextra -Wno-implicit-fallthrough -o pb/boot/pb/main.o -c c/main.c
/usr/bin/clang -arch arm64 -O2 -o pb/bin/pb/scheme pb/boot/pb/main.o pb/boot/pb/libkernel.a -L/opt/local/lib -liconv -lm -lncurses -lz -llz4
: pb/bin/pb/scheme
running pb/bin/pb/scheme to build pb/s/cmacros.so
exec failed
failed
 in build-one
 in loop
 in module->hash
make: *** [build] Error 1

pb/bin/pb/scheme has been compiled for arm64; of course it cannot be executed on an x86_64 machine:

% file pb/bin/pb/scheme
pb/bin/pb/scheme: Mach-O 64-bit arm64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE>

[mflatt]

You're right that BUILDING needs more instructions, at a minimum, and probably there should be more direct support in the configure and make files.

Here's a recipe that should work:

• Build for your current architecture with ./configure and make (but no need to install).
• Build tarm64osx boot files with make tarm64osx.boot.
• Re-configure with ./configure -m=tarm64osx CFLAGS="-arch arm64" --prefix=/tmp/out, replacing /tmp/out with a place where you'd like the result to end up.
• Build with make kernel, which builds just the C-implemented kernel for Arm64 without trying to run it. (This will also take advantage of the fact that the earlier make created bin/zuo, and that one will be used.)
• Use make install to gather everything into (your replacement for) /tmp/out.

This recipe should work in general, except that more work may be needed to select the right C compiler in the middle step, depending on how the target platform relates to the current one.

[ryandesign]

Thanks, I'm trying that out.

We are also currently using the configure args --threads --libkernel LZ4=-llz4 ZLIB=-lz. Should I now be specifying those in the first configure invocation, the second one, or both?

[mflatt]

The second configure is the relevant one for those flags, since I think those are meant to configure the result. They would be fine to use for the first ./configure to build for the current machine, since you have those libraries installed universal.