Setting up an iOS cross-compiler (hosted on macOS)

[fxcoudert]

I am documenting here my notes on how to use this branch to set up a cross-compiler to aarch64-apple-darwin (ideally iOS, not macOS) from x86_64-apple-darwin. I hope you don't mind Iain :) The goal is actually to produce an iOS cross-compiler.

I managed to get something working with:

../gcc/configure --prefix=/tmp \
  --build=x86_64-apple-darwin20 --target=aarch64-apple-darwin20 \
  --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl \
  --with-native-system-header-dir=/usr/include \
  --with-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk \
  AS_FOR_TARGET=/usr/bin/as LD_FOR_TARGET=/usr/bin/ld \
  NM_FOR_TARGET=/usr/bin/nm RANLIB_FOR_TARGET=/usr/bin/ranlib \
  AR_FOR_TARGET=/usr/bin/ar LIPO_FOR_TARGET=/usr/bin/lipo \
  DSYMUTIL_FOR_TARGET=/usr/bin/dsymutil \
  --disable-multilib

Some notes on this:

• it's important to set build and not host, as GCC otherwise thinks it's setting up a Canadian cross (which fails)
• the sysroot for iOS is the iPhoneOS.sdk SDK (weird name…)
• the *_FOR_TARGET are important, as they are not automatically detected by gcc
• the *_FOR_TARGET need to have an absolute path, otherwise the configure script will discard them (I don't fully understand why, and this cause a lot of pain to figure out)
• I don't understand why --disable-multilib would be needed, but without it, GCC tries to build two variants of aarch64-apple-darwin20: a normal one and a lp64 one. This feels like a bug, really.

---

What is the result? Well, the compiler builds and most of the runtime libraries do, as well.

• One limitation is a ld warning:

ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_notify.dylib) built for iOS

I'm not sure where in GCC we're creating macOS-specific code, but it may be from passing the linker the -mmacosx-version-min= option in various places.

• libgfortran fails to build with implicit declaration of function ‘getentropy’, because on the iOS SDK, getentropy exists (thus configure finds it in the system libraries) but it's not declared (and you're not supposed to use it, it's considered a private API). If I bypass this, the build succeeds.
• libssp fails (not that I care much) with

Undefined symbols for architecture arm64:
  "_syslog$DARWIN_EXTSN", referenced from:
      _fail in ssp.o

• libgcc, libatomic, libgomp, libobjc-gnu and libstdc++ all build fine 🍸

---

_2023 edit: DSYMUTIL_FOR_TARGET=/usr/bin/dsymutil is also needed now._

[iains]

there's some old statement about "walking before running" ;-)

There are a number of things we never implemented in GCC because we had no iPhoneOS/iOS/tvOS/watchOS support - e.g. min versions etc. My guess is that those will affect the way in which SDK headers get used. Probably not rocket science to add what's needed (but we'll have to add the validation of xxOS versions in place of that used for macOS).

(conjecture) --disable-multilib is probably needed because the versioning is all "unexpected" and thus the rules for what to use and what to include are probably wrong (or at least somewhat off).

[when starting on the Arm64 port] I went through an exercise of comparing what the clang driver produced in response to various permutations of --xxxxx-version-min, archxxx, etc. .. and it's somewhat complex ..

it's likely that the code-gen is not too much wrong but the configury and packaging will need working through...

[iains]

of course, we have no support for Arm iPhoneOS at all (no Mach-O backend stuff for the Arm port) - so you would have to ensure that the configury did not try to do that.

[fxcoudert]

@iains what would be the best way, in your opinion, for GCC to make a difference between iOS and macOS on ARM, both being aarch64-apple-darwin triplets?

I know “walk before you run”, but to be honest I think for some limited use cases, we're not far from having a Fortran compiler for iOS, which is something people have been asking for a really long time. It would enable things like scipy and other scientific software. The limited use case could be “generating object files and static libraries” to link in a bigger project, for example.

[iains]

(long[er] term)

That is somewhat confusing - actually, IIRC, clang uses -target xxxxx-apple-darwin == iOS and xxxxx-apple-macos[x]NN.MM == MacOS .. and then some things extra for tvOS and watchOS .. I'd need to run through the exercise again.

It's a plan to get the darwin port to recognise "-target" and try to harmonise at least the user-facing command line options with those of clang.

=== (short[er] term)

I think that if we can let the backend know we're targeting iOS, probably we don't need to do too much heavy lifting.

1) Implement -miphoneos-min-version (or whatever is the most reasonable one for iOS)

• including the various checks on validity

2) go through the Darwin specs and add in the ones reflecting the right actions (especially making sure that the assembler and linker are called correctly)

3) See if we need some configure-level change to deal with this (I hope not)

• however, I expect that we need something like darwin20 == macOS 11 , == iOS XX == tvOS YY == watchOS ZZ (for whatever versions are deemed useful)

---

My concern is the the backend is experimental and has known issues; you understand that, I understand that - but once published for end users , be prepared for complaints that "it don't work".

[iains]

sigh if only all the end users cared enough to fund at least one full-time Darwin engineer .. that would still be hardly enough .. but at least a step in the right direction (the Linux distros have a team of toolchain engineers, after all).

[fxcoudert]

@iains I know I've been trying to figure out ways we could get funding to improve the state of things… don't know good ways, it seems some open source is now funded through foundations (like Gates for some machine learning toolkits, etc)

[isuruf]

-target xxxxx-apple-darwin == iOS

In clang, xxxxx-apple-darwin implies macOS. See for eg: https://github.com/llvm/llvm-project/blob/62ec4ac90738a5f2d209ed28c822223e58aaaeb7/llvm/unittests/ADT/TripleTest.cpp#L1184-L1186 xxxxx-apple-darwin20 is internally xxxxx-apple-macosx11.0. You need xxxxx-apple-ios for iphoneOS.

[holzschu]

In my experience xxxxx-apple-darwin for clang implies both macOS an iOS. That is a bother, as x86_64-apple-darwin means both MacOS and the iOS simulator, and arm64-apple-darwin means both MacOS running on the new processor and iOS running on a phone. xxxxx-apple-ios also works, but it is less often used.

[iains]

there are two places that count ;

1. the user-facing command lines - (which go to the driver) which accept a combination of -arch and -mxxxxxx-min-version etc, etc. and produces ....

2. ... an internal target triple which is passed to clang -cc1 / clang -cc1as

GCC also has a driver (that's what you get when you type gcc ...) and Darwin has driver code to normalise some of these things; In particular, to ensure that cc1 gets sensible options and that __ENVIRONMENT_MACOSX_VERSION_MIN__ gets set to a valid/sensible value. That also means that we can select appropriate values to pass on to the assembler (which for many cases actually calls back to the Xcode as which, in turn, calls clang -cc1as) and the linker.

We have not, so far, implemented the equivalent checks and balances for iOS. So I'd say "all bets are probably off" about how the SDK headers are interpreted and values passed to the assembler / linker.

You might be lucky - but I expect that currently we're lying to both the assembler and linker (claiming to be macOS when you really wanted iOS).

[demhademha]

An update of progress by myself, I have successfully been able to compile gmp, mpc, mpfr for iOS and made sure they work. I've also done most of the other gcc dependencies including binutils

[demhademha]

I am building on linux, so things may be a little different: So far, I've managed to get configure and make to run, but when configuring gcc itself, it fails with the following log as the c compiler is unable to work: https://paste.debian.net/1185250/

What is bizarre is that configure does not respect the host of the initial configuration (which was aarch64-apple-darwin) rather, it sees the host as the build system

Edit: manually going to the Build folder and rerunning make works

[demhademha]

I am documenting here my notes on how to use this branch to set up a cross-compiler to aarch64-apple-darwin (ideally iOS, not macOS) from x86_64-apple-darwin. I hope you don't mind Iain :) The goal is actually to produce an iOS cross-compiler.

I managed to get something working with:

../gcc/configure --prefix=/tmp \

  --build=x86_64-apple-darwin20 --target=aarch64-apple-darwin20 \

  --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl \

  --with-native-system-header-dir=/usr/include \

  --with-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk \

  AS_FOR_TARGET=/usr/bin/as LD_FOR_TARGET=/usr/bin/ld \

  NM_FOR_TARGET=/usr/bin/nm RANLIB_FOR_TARGET=/usr/bin/ranlib \

  AR_FOR_TARGET=/usr/bin/ar LIPO_FOR_TARGET=/usr/bin/lipo \

  --disable-multilib

Some notes on this:

• it's important to set build and not host, as GCC otherwise thinks it's setting up a Canadian cross (which fails) on linux, this is not possible as when configure is testing the compiler, it gets an exec format error. Somehow, we need to tell gcc to use --host and --target for ios

[fxcoudert]

@demhademha the procedure discussed here only works for a macOS host machine, because it's using the multi-arch Apple compiler/assembler/linker.

[Green-Cat]

Thank you for the work so far.

How did you get the cross-compiler to actually target iOS after it is compiled? Since I haven't found an explicit way to target iOS rather than macOS, I get: clang: warning: using sysroot for 'iPhoneOS' but targeting 'MacOSX' [-Wincompatible-sysroot] and ld: building for iOS, but linking in object file built for macOS, file 'cfortran_test.o' for architecture arm64

Is there some flag to specify that I am building for iOS? I am currently just using "-arch arm64", as gfortran does not seem to support "-target".

[holzschu]

You need both -arch arm64 and -miphoneos-version-min=14.0. I've had to edit some files to make sure the compiler inserts -miphoneos-version-min=14.0 instead of -mmacosx-version-min=11.0. The result is a compiler that can only compile for iOS, which is fine for me but not for distribution.

As @fxcoudert said, the command line to compile is a bit large:

# !!! make sure it's /usr/local/bin/gcc and not /usr/bin/gcc 
../gcc_arm/configure --build=x86_64-apple-darwin20 --target=aarch64-apple-darwin20 \
--prefix="/usr/local" \
--disable-multilib \
--disable-libssp \
--disable-libstdcxx \
--enable-languages=c,fortran \
--with-ld="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" \
AR="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar" \
LD="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" \
AS="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as" \
RANLIB="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib" \
NM="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm" \
AR_FOR_TARGET="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar" \
LD_FOR_TARGET="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" \
AS_FOR_TARGET="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as" \
RANLIB_FOR_TARGET="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib" \
REAL_LD_FILE_NAME="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" \
NM_FOR_TARGET="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm" \
LIPO="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo" \
LIPO_FOR_TARGET="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo" \
CFLAGS_FOR_TARGET="-miphoneos-version-min=14.0" \
CXXFLAGS_FOR_TARGET="-miphoneos-version-min=14.0"\
LDFLAGS_FOR_TARGET="-miphoneos-version-min=14.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk -lSystem"\
--with-native-system-header-dir=/usr/include \
--with-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk

[asdfugil]

Just a note: There's also iPhone Simulator, Watch Simulator, Apple TV Simulator, Mac Catalyst, DriverKit as well as bridgeOS targets.

From ld(1) on iOS

-platform_version platform min_version sdk_version
             This is set to indicate the platform, oldest supported version of that
             platform that output is to be used on, and the SDK that the output was
             built against.  platform is a numeric value as defined in <mach-
             o/loader.h>, or it may be one of the following strings:
             • macos
             • ios
             • tvos
             • watchos
             • bridgeos
             • mac-catalyst
             • ios-simulator
             • tvos-simulator
             • watchos-simulator
             • driverkit
             Specifying a newer min or SDK version enables the linker to assume fea‐
             tures of that OS or SDK in the output file. The format of min_version
             and sdk_version is a version number such as 10.13 or 10.14

[Torrekie]

_syslog$DARWIN_EXTSN was not exist on iOS, but you can find the source from syslog

[asdfugil]

Note that for the 64-bit version watchOS have 32-bit pointers on a 64-bit CPU (much like x32 in this regard), and is called arm64_32, which is a different ABI.

[meow464]

I'm trying to follow the instructions in the OP on macOS Monterey x86 but it fails at libgcc_s.dylib:

libgcc_s.dylib error

``` /Users/tuco/Projects/gcc-darwin-arm64-build/./gcc/xgcc -B/Users/tuco/Projects/gcc-darwin-arm64-build/./gcc/ -B/tmp/aarch64-apple-darwin20/bin/ -B/tmp/aarch64-apple-darwin20/lib/ -isystem /tmp/aarch64-apple-darwin2 0/include -isystem /tmp/aarch64-apple-darwin20/sys-include -O2 -g -O2 -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wold-s tyle-definition -isystem ./include -mmacosx-version-min=10.8 -mmacosx-version-min=11.0 -fno-common -mmacosx-version-min=11.0 -g -DIN_LIBGCC2 -fbuilding-libgcc -fno-stack-protector -dynamiclib -nodefaultlibs - install_name @rpath/libgcc_s.1.1.dylib -single_module -o ./libgcc_s.dylib -Wl,-exported_symbols_list,libgcc.map -compatibility_version 1 -current_version 1.1 -nodefaultrpaths -g -O2 -B./ _muldi3_s.o _negdi2_s.o _l shrdi3_s.o _ashldi3_s.o _ashrdi3_s.o _cmpdi2_s.o _ucmpdi2_s.o _clear_cache_s.o _trampoline_s.o __main_s.o _absvsi2_s.o _absvdi2_s.o _addvsi3_s.o _addvdi3_s.o _subvsi3_s.o _subvdi3_s.o _mulvsi3_s.o _mulvdi3_s.o _ne gvsi2_s.o _negvdi2_s.o _ctors_s.o _ffssi2_s.o _ffsdi2_s.o _clz_s.o _clzsi2_s.o _clzdi2_s.o _ctzsi2_s.o _ctzdi2_s.o _popcount_tab_s.o _popcountsi2_s.o _popcountdi2_s.o _paritysi2_s.o _paritydi2_s.o _powisf2_s.o _po widf2_s.o _powixf2_s.o _powitf2_s.o _mulhc3_s.o _mulsc3_s.o _muldc3_s.o _mulxc3_s.o _multc3_s.o _divhc3_s.o _divsc3_s.o _divdc3_s.o _divxc3_s.o _divtc3_s.o _bswapsi2_s.o _bswapdi2_s.o _clrsbsi2_s.o _clrsbdi2_s.o _ fixunssfsi_s.o _fixunsdfsi_s.o _fixunsxfsi_s.o _fixsfdi_s.o _fixdfdi_s.o _fixxfdi_s.o _fixunssfdi_s.o _fixunsdfdi_s.o _fixunsxfdi_s.o _floatdisf_s.o _floatdidf_s.o _floatdixf_s.o _floatundisf_s.o _floatundidf_s.o _floatundixf_s.o _fixsfti_s.o _fixdfti_s.o _fixxfti_s.o _fixtfti_s.o _fixunssfti_s.o _fixunsdfti_s.o _fixunsxfti_s.o _fixunstfti_s.o _floattisf_s.o _floattidf_s.o _floattixf_s.o _floattitf_s.o _floatuntisf_s.o _fl oatuntidf_s.o _floatuntixf_s.o _floatuntitf_s.o _divdi3_s.o _moddi3_s.o _divmoddi4_s.o _udivdi3_s.o _umoddi3_s.o _udivmoddi4_s.o _udiv_w_sdiv_s.o sync-cache_s.o sfp-exceptions_s.o addtf3_s.o divtf3_s.o eqtf2_s.o g etf2_s.o letf2_s.o multf3_s.o negtf2_s.o subtf3_s.o unordtf2_s.o fixtfsi_s.o fixunstfsi_s.o floatsitf_s.o floatunsitf_s.o fixtfdi_s.o fixunstfdi_s.o floatditf_s.o floatunditf_s.o fixtfti_s.o fixunstfti_s.o floatti tf_s.o floatuntitf_s.o extendsftf2_s.o extenddftf2_s.o extendhftf2_s.o trunctfsf2_s.o trunctfdf2_s.o trunctfhf2_s.o fixhfti_s.o fixunshfti_s.o floattihf_s.o floatuntihf_s.o heap-trampoline_s.o enable-execute-stack _s.o emutls_s.o libgcc.a -lSystem ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libcache.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libcommonCrypto.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libcompiler_rt.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libcopyfile.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libcorecrypto.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libdispatch.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libdyld.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libmacho.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libremovefile.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_asl.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_blocks.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_c.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_collections.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_configuration.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_containermanager.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_coreservices.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_darwin.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_dnssd.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_featureflags.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_info.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_kernel.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_m.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_malloc.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_networkextension.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_notify.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_platform.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_product_info_filter.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_pthread.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_sandbox.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_symptoms.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libsystem_trace.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libunwind.dylib) built for iOS ld: warning: building for macOS, but linking in .tbd file (/usr/lib/system/libxpc.dylib) built for iOS Undefined symbols for architecture arm64: "_pthread_jit_write_protect_np", referenced from: ___builtin_nested_func_ptr_created in heap-trampoline_s.o ld: symbol(s) not found for architecture arm64 collect2: error: ld returned 1 exit status make[2]: *** [libgcc_s.dylib] Error 1 make[1]: *** [all-target-libgcc] Error 2 make: *** [all] Error 2 ```

The only difference from OP is that I added --enable-languages=c,fortran at the end of the configure command line.

Any ideas?

EDIT: pthread_jit_write_protect_np is not supported on iOS. I simply commented it out in libgcc/config/aarch64/heap-trampoline.c (blamed commit) but ideally it would use a macro.