Support Apple app store bitcode

[brson]

Bitcode is the future of Apple's app distribution and we don't support it now. Doing so is tricky because Apple choses whatever rev of LLVM they want and upgrade it per their whims. We can't couple our LLVM to theirs because we have our own needs and can't be forced to upgrade whenever Apple decides.

Here are some options:

• We ship an entirely separate toolchain for targetting Apple bitcode. This is quite ugly.
• We change rustc to dynamically load LLVM, create a rust-llvm package and an optional rust-llvm-apple package, and have rustc load the apple LLVM to deal with their bitcode.
• We create a rustc_llvm_apple crate and just link two entire LLVMs into rustc at all times. This probably would not fly
• We create a build of rustc called rustc-apple and ship it in an optional rustc-apple package. It is exactly like rustc except instead of linking to rustc_llvm it links to rustc_llvm_apple. When rustc gets a request to emit bitcode it just defers completely to the rustc-apple binary. I actually fancy this solution quite a bit.
• We could investigate a bitcode sanitizer to translate our bitcode to theirs. I think the likelihood of success here is low because maintenance. I believe RenderScript does this.

I think the LLVM dynamic loading and defer-to-alternate-rustc solutions are most promising.

cc https://users.rust-lang.org/t/ios-rust-integration/6928/4

cc @bluejekyll

[nagisa]

We change rustc to dynamically load LLVM, create a rust-llvm package and an optional rust-llvm-apple package, and have rustc load the apple LLVM to deal with their bitcode.

Dynamically loading LLVM is certainly not an option. The LLVM API is less stable than its bitcode and, if we build against LLVM x.y, then having LLVM x.z instead of x.y will almost certainly break rustc. Just link to the system LLVM statically (like --llvm-root already does).

We create a rustc_llvm_apple crate and just link two entire LLVMs into rustc at all times. This probably would not fly

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

We could investigate a bitcode sanitizer to translate our bitcode to theirs. I think the likelihood of success here is low because maintenance. I believe RenderScript does this.

Okay… so, the thing is that LLVM bitcode format is kinda stable between versions of LLVM. Does apple do something unusual to make that not true, or is the bitcode format used not LLVM altogether (i.e. has apple specific stuff)?

---

I feel like I heard all the way back in 2013-ish that using LLVM bitcode as distribution format is stupid. I certainly agree; does distributing native binary libraries not work anymore for iOS?

[nagisa]

I’m also very interested in what would happen when Apple upgrades to LLVM version with incompatible bitcode format internally and people still compile stuff with old compilers.

[bluejekyll]

I experimented briefly with this, i.e. output bitcode during cargo builds, then tried to create a static library with the bitcode included. When I tried to llvm-link against the bitcode library, I got an incompatible LLVM version from the bitcode.

I don't have an easy test to reproduce this, but my guess is there is just a dumb version check that denies linking between different LLVM versions? Or, I just did something entirely wrong. I will try to come up with a test case when I have some time to look at this again.

[sanxiyn]

It would be useful to have exact error message from llvm-link.

[nagisa]

@bluejekyll LLVM has a number of bitcode files in its test directory. Tests against these bitcode files are run continuously (llvm-dis-3.8/opt-3.8 understands a 3 year old bitcode file from 3.2 just fine, for example), so it must be some apple stuff.

[ricky26]

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

This is somewhat easier said than done - as far as I'm aware, the only blessed apple-llvm version for App Store uploads is the one shipped with the current Xcode. This also potentially means maintaining the LLVM bindings for two LLVM versions (not necessarily two neighbouring minor versions, either). I don't think it's valid to just use an old version of apple-llvm.

I’m also very interested in what would happen when Apple upgrades to LLVM version with incompatible bitcode format internally and people still compile stuff with old compilers.

I think they avoid this by only allowing the latest Xcode to submit apps (and I'm pretty sure it's burned into the output image which LLVM version you used).

IIRC, the bitcode is packaged for each architecture, since cross-architecture bitcode isn't generated by clang (I think that's an anti-goal of clang and bitcode in general). It's stored in a section for each object file so at the very least it's duplicated. That could be part of why someone mentioned that bitcode might not be the best way of going about this.

I feel a bit like all of the recommended solutions are somewhat icky. The least messy way I can think of is to allow different targets to overload the codegen behaviour and have the Apple codegen path be in a dynamic crate. (Which would just be the regular codegen path compiled against apple-llvm.)

Since this bug mentions the App Store, is it worth talking about the exception handling story here? (i.e. panic=abort is strictly required at the moment.)

[bluejekyll]

LLVM has a number of bitcode files in its test directory. Tests against these bitcode files are run continuously (llvm-dis-3.8/opt-3.8 understands a 3 year old bitcode file from 3.2 just fine, for example), so it must be some apple stuff.

@nagisa thanks for letting me know this. It gives me hope that there might still be a solution here, and that I was probably doing something wrong.

[nagisa]

@ricky26 good points.

as far as I'm aware, the only blessed apple-llvm version for App Store uploads is the one shipped with the current Xcode.

Isn’t Xcode LLVM on apple the same thing as system LLVM? I meant the Xcode LLVM then. We would have to make sure xcode is always the most recent version when shipping rustc trains.

Of course the way apple does things rules out us producing a valid apple bitcode with an old version of rustc and essentially forces us throw all the benefits our stability story provides out the window, and I don’t see any way this could be fixed.

This also potentially means maintaining the LLVM bindings for two LLVM versions

We already maintain¹ support for LLVM versions 3.7 through 3.9 (and potentially trunk). As long as Xcode’s LLVM is not some ancient version, I think we’re good in that regard. If Xcode LLVM is really some ancient/custom/etc version, then I don’t see us being able to support this feature at all, then. Especially since we have no option of sending patches to that LLVM in order to add the features we need. I also wouldn’t want to lock rustc into supporting 3.7 forever in case Apple decided to not update Xcode LLVM until 2038.

¹: however, if rustc was built against LLVM x.y, it must be linked to LLVM x.y exactly.

[brson]

Dynamically loading LLVM is certainly not an option. The LLVM API is less stable than its bitcode and, if we build against LLVM x.y, then having LLVM x.z instead of x.y will almost certainly break rustc. Just link to the system LLVM statically (like --llvm-root already does).

@nagisa The C++ API is unstable, but we use the C API and have had a lot of success supporting multiple versions of LLVM at once. I don't see the difference in terms of API support.

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

We could just blanket ship Apple's LLVM for all apple platforms, but this means coupling our LLVM to Apples for even desktop machine code generation, and precludes the option of supporting iOS bitcode on non-apple hosts.

Okay… so, the thing is that LLVM bitcode format is kinda stable between versions of LLVM. Does apple do something unusual to make that not true, or is the bitcode format used not LLVM altogether (i.e. has apple specific stuff)?

The bitcode format is not stable between versions.

does distributing native binary libraries not work anymore for iOS?

It does work today. It is not the preferred method and it's not obvious it will continue to be supported.

This also potentially means maintaining the LLVM bindings for two LLVM versions (not necessarily two neighbouring minor versions, either). I don't think it's valid to just use an old version of apple-llvm.

@ricky26 We successfully maintain compatibility between several versions of LLVM. As long as Apple's and ours don't drift too far apart it should be doable, but there's always the risk of such great breakage that the divide can't be crossed, and I know there are major API changes coming.

[bluejekyll]

As long as Xcode’s LLVM is not some ancient version, I think we’re good in that regard.

From this page https://gist.github.com/yamaya/2924292:

clang-700.0.72 => LLVM 3.7.0
clang-700.1.76 => LLVM 3.7.0
clang-700.1.81 => LLVM 3.7.0
clang-703.0.29 => LLVM 3.8.0
clang-703.0.31 => LLVM 3.8.0

[nagisa]

The C++ API is unstable, but we use the C API and have had a lot of success supporting multiple versions of LLVM at once. I don't see the difference in terms of API support.

That is not true. We have (quite large!) a number of bindings to C++ APIs in form of rustllvm. There’s a number of cases where we compile that wrapper depending on the version of LLVM compiled against. In case version of LLVM used and compiled against does not match, you’ll get dynamic linker errors, or worse, run into issues at runtime.

precludes the option of supporting iOS bitcode on non-apple hosts.

If Apple does not want to take bitcode generated by anything else than their fork of LLVM, then I don’t see how we could do anything here short of maintaining a similar fork and reverse-engineering their internal patches.

The bitcode format is not stable between versions.

Sure, but it is quite fair to assume¹ that bitcode between various revisions of LLVM known as 3.7.0, for example, is sufficiently compatible for purposes of generating bitcode for consumption by another build of LLVM from 3.7.0 series. Its certainly better than linking to libLLVM dynamically.

¹: especially given that bitcode from 3.2 series is still compatible with 3.8 LLVM, even if its a very small specimen.

[comex]

Some notes:

• Xcode doesn't even ship with a linkable libLLVM (static or dynamic), only a libclang.dylib which statically links to LLVM.
• Apple does ship the source to their 'clang' (including LLVM) under 'Developer Tools' on https://opensource.apple.com, but that site tends to take forever to be updated.
• The story may be better with Swift, which has its own fork of LLVM with release tags - but they may not correspond perfectly to what's shipped with Xcode. (It's possible to use open-source toolchain snapshots with Xcode, but projects built with such toolchains can't be submitted to the App Store.)

[aturon]

cc @rust-lang/compiler

[mitsuhiko]

Would be curious to hear how other programming languages plan on dealing with this. In particular mono and go.

[ricky26]

Unity's answer to this problem is il2cpp - building all of their IL assemblies into C++ code.

Relevant golang bug: https://github.com/golang/go/issues/12682; the suggestion there seems to be that they could use the LLVM go toolchain (which isn't as featured as the standard go toolchain).

All in all, the story for bitcode support outside of Apple is poor.

[mitsuhiko]

Mono proper goes via apple LLVM it would appear: http://tirania.org/blog/archive/2015/Sep-02.html

[mitsuhiko]

One stumbling block will be that you cannot carry inline assembly in bitcode :(

For the mono story I had a quick exchange with Miguel de Icaza about what Mono does for the curious: https://twitter.com/mitsuhiko/status/769458873237434368

[comex]

@mitsuhiko You can have inline assembly in bitcode on iOS and tvOS, but not watchOS, for some reason.

[russellmcc]

Any movement on this? I don't feel at all good about using Rust on iOS without a plan for supporting bitcode. Apple has a history of making optional things like this non-optional fairly suddenly, and indeed bitcode is already required on watchOS and tvOS.

I feel a bit like all of the recommended solutions are somewhat icky. The least messy way I can think of is to allow different targets to overload the codegen behaviour and have the Apple codegen path be in a dynamic crate. (Which would just be the regular codegen path compiled against apple-llvm.)

This approach (by @ricky26) seems to be the most natural to me as a rustc user.

[alexcrichton]

I don't believe anything has changed on this recently to my knowledge at least. With LLVM's recent announcement about versioning they indicated that bitcode should (I think) always be loadable by future versions of LLVM. That may mean that this issue is "solved" on a fundamental level, but it'd still require a more ergonomic interface to extract all the bitcode.

[adrianbrink]

Are there any updates on this?

[samkhawase]

This commenter on HackerNews has succeeded in using Bitcode generated from Rust on macOS and iOS. The thread has some detailed information on how to enable bitcode for rust binaries, which sounds like a great news!

https://news.ycombinator.com/item?id=14305084

[comex]

As the commenter in question, quick notes:

• I used -C lto --emit llvm-bc to get rustc to emit one .bc file containing the current crate and all dependencies. This works, but is essentially a hack; in particular, it doesn't work for C dependencies, including jemalloc (though that isn't used on iOS anyway). It would be better if rustc properly supported emitting Mach-Os with "embedded bitcode"; you can look at the clang source to see how it's done.

• If you want to try it and don't mind the hack, it does seem to 'just work', except for the version issue:

• The main practical obstacle is that Rust syncs with LLVM trunk more frequently than Xcode, which seems to only do so yearly. Newer LLVM versions can load old bitcode, but not the other way around, so you have to either use an old version of rustc or build the latest against an old LLVM. (Actually, rustc 1.17 seems to still work with Xcode 8.x, but not nightly since the LLVM 4.0 upgrade; that's just coincidence though.)

• Ideally Rust would ship official binaries built against a suitable LLVM version. In practice, it seems to be fine to use the correct version of stock LLVM, but if you do want to use Apple's fork:

  • As I said in a previous comment, Xcode doesn't ship LLVM binaries in any linkable form, even dynamically; only clang and libclang.dylib (which exports the Clang API but not the LLVM one).
  • But, as I also said, the source code usually goes up on opensource.apple.com (under Developer Tools -> clang; the archive includes all of LLVM), just somewhat inconsistently/after a delay. However, especially given the expanded backwards compatibility guarantees noted by @alexcrichton, it may not be the end of the world to not always be up to date. Currently, the most recent source release is for Xcode 8.2.1, whereas the latest is 8.3.2.

[comex]

(As a test, I just tried compiling Rust against the Xcode 8.2.1 Clang. rustllvm fails to build, because its nest of #if LLVM_VERSION_GE(..) conditionals, which are meant to allow it to compile against both older and newer LLVM, gets confused by this branch - which claims to be LLVM 3.9svn, but API-wise is somewhere in between 3.8 and 3.9. Probably not worth fixing, since Xcode is almost due for its next yearly refresh anyway.)

[regexident]

Has any progress been made on this front? Alternatively would you (core team) consider this ordeal an issue with a solid permanent fix in sight or do you expect this version hopping thing to remain for the foreseeable future?

I'm considering implementing parts of an iOS app in Rust for its superior type system and low-level semantics and would rather not be stuck with non-bitcode or regular hacks/delays whenever there's an update for either Xcode or rustc.

[arielb1]

@regexident

I think we are going to be implementing swappable LLVM versions for other reasons. If that is implemented, we shouldn't have a problem shipping separate trans for Apple, wasm and everything else.

cc #45684

[burdges]

Isn't Apple's bitcode still really dangerous ala https://medium.com/@FredericJacobs/why-i-m-not-enabling-bitcode-f35cd8fbfcc5 Impossible to verify builds. Issues for cryptography. etc.

[regexident]

@burdges For certain apps this absolutely is a problem.

But for watchOS and tvOS Apple does in fact require Bitcode for App Store submissions. You don't have a choice. Other than "don't do crypto or critical stuff on those platforms". One also has to fear Apple enforcing Bitcode for iOS at some point in the future. I'd rather not have a bricked product at that point.

[michaeleiselsc]

I tried building with Rust 1.24.0 for iOS using the -C lto --emit=llvm-bc trick, but the linker gave the following errors:

Undefined symbols for architecture x86_64:
  "_backtrace_create_state", referenced from:
      std::sys_common::gnu::libbacktrace::init_state::h686c3e443c712b0f in Logger(x86_64.o)
  "_backtrace_syminfo", referenced from:
      std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::h598a932d5bb0d80b in Logger(x86_64.o)
      core::iter::iterator::Iterator::position::_$u7b$$u7b$closure$u7d$$u7d$::hbf03153d55553502 in Logger(x86_64.o)
  "_backtrace_pcinfo", referenced from:
      std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::h598a932d5bb0d80b in Logger(x86_64.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Any ideas what might be the issue?

Edit: it appears that I can at least suppress that issue by wrapping all the code in each public extern fn with AssertUnwindSafe

[comex]

libbacktrace is a C library which is included in the Rust source tree:

https://github.com/rust-lang/rust/tree/master/src/libbacktrace

You'll have to compile that as bitcode somehow.

[michaeleiselsc]

I'm confused, because I can only find a libbacktrace built for my laptop, but not for the phone OS's. How does rust insert those symbols into the stdlib? Is there an easier way to do this, by somehow disabling backtrace functionality? I removed the RUST_BACKTRACE env var but that didn't help. I've read that you can compile rustc without backtraces, but I was hoping for something easier, e.g. in the cargo.toml file of my project

[comex]

Fundamentally you want to build the rustc source tree while telling it to pass -fembed-bitcode to the C compiler. Ideally you'd just be able to add it to CFLAGS; unfortunately, I just tried that, and it didn't work properly due to two issues with gcc-rs (used by rustc's build system to build C dependencies). However, I got it to work with a bit of a hacky procedure:

cd /tmp/build
cat >ccwrap.py <<END

#!/usr/bin/env python
import os, sys
os.execv('/usr/bin/xcrun', ['clang', '-fembed-bitcode'] + [arg for arg in sys.argv[1:] if arg not in {'-ffunction-sections', '-fdata-sections'}])

END

chmod 755 ccwrap.py
ln -s /usr/bin/ar ar
export CC_aarch64_apple_ios=`pwd`/ccwrap.py
/usr/src/rust/configure --target=aarch64-apple-ios
make

This should get you a libbacktrace.a (in ./build/aarch64-apple-ios/native/libbacktrace/.libs/) with bitcode embedded. You will still have to manually link it. (It does, but this also ends up getting statically linked into libstd.rlib, which is what you really should be linking.) As I mentioned upthread, the 'proper' solution is for rustc to embed bitcode in the __LLVM,__bitcode section of what are otherwise native libraries, the same way clang does: then there would be no need to manually compile and link .bc files.

(Actually, since the error you quoted looks like a normal linker error, I think you could get past it just by linking a regular, no-bitcode-embedded libbacktrace.a. But at best that would result in an output file with broken embedded bitcode.)

The aforementioned issues:

• gcc-rs passes -ffunction-sections and -fdata-sections with no way to turn them off; clang complains about these when combined with -fembed-bitcode (even if -fno-function-sections is passed later).

I filed an issue report, and as a workaround, the above procedure uses a wrapper script that strips out those arguments. But that caused a second problem:

• gcc-rs tries to find ar in the same directory as CC_aarch64_apple_ios (!?); manually setting AR_aarch64_apple_ios had no effect

I couldn't reproduce this with the latest cc-rs (renamed from gcc-rs), so I didn't file an issue. As a workaround, the above procedure symlinks ar into the same directory.

[michaeleiselsc]

Thanks for the suggestions! I actually just needed bitcode for my own code, not the backtrace stuff, so I just used the compiled libbacktrace without any bitcode embedding

[michaeleiselsc]

With the hack I tried, I'm now hitting another issue: dsymutil segfaults when I try to create a dSYM for it. Normal execution without a dSYM works fine, but dSYM creation is broken for some reason. I removed my libbacktrace.a, replaced it with empty placeholder functions, and things worked fine, so it seems that there's an issue with my rust lib.

[comex]

That sounds like a bug in dsymutil, which should be reported upstream to LLVM. Can you try running dsymutil under LLDB and posting a backtrace of the crash? Or to get better debug info, you could try building LLVM in debug mode and reproducing it with that.

(Note that the binary in a stock LLVM installation is called llvm-dsymutil, but these days Xcode's dsymutil is just a symlink to llvm-dsymutil. dsymutil used to be a separate, closed source utility, but for a few years that version has been shipped as dsymutil-classic and not used by default.)

[alexcrichton]

@comex out of curiosity, should the cc crate compile everything with -fembed-bitcode? Or if we're using the "official" clang is it ok to avoid embedding bitcode?

(sorry I'm not super familiar with ios best practices!)

For Rust code itself, would it be best if we just emit bytecode instead of object files? Is that something that Clang/the linker can understand?

[comex]

@alexcrichton

should the cc crate compile everything with -fembed-bitcode? Or if we're using the "official" clang is it ok to avoid embedding bitcode?

Using the official clang doesn't obviate the need for embedded bitcode. Rather, the bitcode gets included - in addition to native code - in the app archive uploaded to Apple, and then Apple can recompile it on the server side, e.g. to optimize for different microarchitectures.

It would probably make sense to pass -fembed-bitcode by default on iOS/watchOS/tvOS. If so, when building in debug mode, -fembed-bitcode-marker should be passed (instead or in addition, doesn't matter); this tells clang to only include a dummy bitcode section rather than any actual bitcode, since embedded bitcode is only needed for final app archives and it slightly speeds up compilation to omit it. Actually, Xcode passes -fembed-bitcode-marker for everything other than building the final app archive, even development builds that happen to have optimizations on - but since cc doesn't have a separate concept of "really final release mode", it should probably just pass it in debug mode. Note that it's safe to embed bitcode even if the final build doesn't require it.

For Rust code itself, would it be best if we just emit bytecode instead of object files? Is that something that Clang/the linker can understand?

Both clang and ld do support raw bitcode files passed as inputs; this is normally used when LTO is enabled (-flto), in which case the .o files clang generates are actually raw bitcode, not Mach-O. (These can still be formed into static libraries using the normal lipo command.) However, this is separate from the "embedded bitcode" format, which consists of a Mach-O with bitcode stuffed into the __LLVM,__bitcode section, in addition to native code in the usual sections. This format is used in object files when LTO is off, as well as in final linked executables or dynamic libraries regardless of the LTO setting. (If LTO is on, the linker is responsible for creating this section; if LTO is off, the linker simply concatenates the __LLVM,__bitcode sections from each object file like it would for any other section, rather than generating a single combined bitcode module.) EDIT: actually, it does something a bit more complicated, generating an xar archive of the bitcode sections and putting them in a section named __LLVM,__bundle; anyway, not something we have to care about.

I think ideally rustc should produce "embedded bitcode" by default on iOS, not raw bitcode. It's not too hard to do, and it avoids breaking incremental compilation - especially in debug mode where you can just produce a dummy bitcode section, ala -fembed-bitcode-marker, but potentially in release mode too, since the linker just stuffs the objects' bitcode sections together rather than doing anything expensive at link time.

Here's how clang generates embedded bitcode:

https://github.com/llvm-mirror/clang/blob/master/lib/CodeGen/BackendUtil.cpp#L1242

Relatively simple - before invoking the backend, it dumps the current LLVM module as bitcode, then stuffs that as binary data into a new global variable (in that same module!), placed in the __LLVM,__bitcode section. It seems like kind of a hacky design (why does the frontend have to do this manually?), but it shouldn't be too hard to reproduce in rustc.

[michaeleiselsc]

@comex that approach makes sense, although I think there are reasons (see #48833 that I just made) for using Apple's clang as the backend, while we're thinking about changes like this. Also, how easy is it to compile ourselves from the embedded bitcode?

[comex]

@michaeleiselsc Sorry, I don't understand what you mean by "compile ourselves from the embedded bitcode".

[michaeleiselsc]

As in, instead of just giving a binary to the App Store with bitcode embedded, there are also benefits to taking the bitcode and building it from that ourselves using Apple clang. Two benefits that I'm interested in are the downstream fixes that Apple clang has and the ability to use clang sanitizers, e.g. the coverage sanitizer.

[alexcrichton]

@comex ok so digging in a bit, it looks like the implementation in rustc is easy enough (relatively), but I've got a few questions:

• Why is -fembed-bitcode-marker a thing? Using clang locally it just emits an empty static into a section. Does that mean the mere presence of the section means something? Does some tool along the road in iOS break if the section isn't there? (but it doesn't actually look inside the section because it's empty?
• Do you know if the __cmdline section is necessary? It looks like -fembed-bitcode also embeds some form of a commandline into the binary. I'm not really sure what we would put here as it's probably not rustc's command line, but does any tool look at this? Does compilation break if -fembed-bitcode=bitcode is used with iOS?
• And finally, is there a reason to ever want to disable this? Or should we just unconditionally enable this for the iOS targets and move on?

[mitsuhiko]

Why is -fembed-bitcode-marker a thing?

I think its used by xcode to indicate that something is supposed to use bitcode but bitcode was not actually compiled. If you try to archive such a project later ld will fail with an error.

[michaeleiselsc]

To the question of whether we want to always enable this, there are definitely some cases we'd want to be careful about. For example, if someone submits it to a beta app distribution platform like HockeyApp, will HockeyApp strip that bitcode section, or will beta testers have to download a much larger binary than they need to?

[comex]

@mitsuhiko Yep, the idea is that the linker can verify that the build is properly set up for bitcode, even if you're currently doing testing builds that don't actually need bitcode.

For example, if you compile a C file without mentioning bitcode, then try to link it with -fembed-bitcode-marker, you get:

$ clang -c -o test.o test.c
$ clang -dynamiclib -o test.dylib test.o -fembed-bitcode-marker
ld: warning: all bitcode will be dropped because 'test.o' was built
without bitcode. You must rebuild it with bitcode enabled (Xcode
setting ENABLE_BITCODE), obtain an updated library from the vendor,
or disable bitcode for this target.

The warning goes away if you pass -fembed-bitcode-marker on the first line as well.

@alexcrichton As for __cmdline… hmm… it seems like the linker requires it to be present (search for "Create bitcode"):

https://opensource.apple.com/source/ld64/ld64-274.2/src/ld/parsers/macho_relocatable_file.cpp.auto.html

But I don't see anything that cares about the actual value. Probably best to put a dummy value there.

[chrisballinger]

Was this issue resolved by merging #48896?

[chrisballinger]

I've been playing around with this on the nightly builds, and it works great until you try to Archive with Xcode. It looks like it's adding fembed-bitcode-marker but not fembed-bitcode to the Rust static library output. It's compiled with cargo lipo --release with:

$ cargo --version
cargo 1.27.0-nightly (af3f1cd29 2018-05-03)
$ cargo lipo --version
cargo-lipo 2.0.0-beta-2
$ rustc --version
rustc 1.27.0-nightly (565235ee7 2018-05-07)

ld: '/Users/chrisbal/Documents/Beach/rust-universal-template/target/universal/release/libexample.a(example_generic-be72fb1769c1779b.example_generic6-152d14edfb6970f54250733c74e59b7.rs.rcgu.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

When I run otool -arch arm64 -l /Users/chrisbal/Documents/Beach/rust-universal-template/target/universal/release/libexample.a | grep bitcode I get a lot of sectname __bitcode, but I don't know how to check if this is actual bitcode and not just markers.

I'm using nested crates so maybe the inner "generic" crate isn't getting the bitcode flags?

edit: here is our repo demonstrating the issue https://github.com/Raizlabs/rust-universal-template/tree/879e7412d729e8963586c5b083d51b09733aec32

[volodg]

@chrisballinger, it works with additional flag, see RUSTFLAGS="-Z embed-bitcode" cargo lipo --release

$ cargo --version
cargo 1.28.0-nightly (f352115d5 2018-05-15)
$ cargo lipo --version
cargo-lipo 2.0.0-beta-2
$ rustc --version
rustc 1.28.0-nightly (952f344cd 2018-05-18)

But I have another compile error only for arm7 arch:

.rs.rcgu.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture armv7

Does anybody know how to fix arm7 arch?

[4lex1v]

Sadly can't get it working as well, getting the same error as @chrisballinger:

ld: '/sandbox/target/universal/release/librgame.a(std-da6dba40351cda22.std3-d36cd881bae00a8b5fc36289b5737f78.rs.rcgu.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I've tried to compile it with RUSTFLAGS="-Z embed-bitcode" cargo lipo --release and setting it in .cargo/config. It adds the flag though:

Executing: "cargo" "build" "--target" "x86_64-apple-ios" "--lib" "--features" "" "--color" "auto" "--release" "--verbose"
   Compiling libc v0.2.42
   Compiling rand_core v0.2.1
     Running `rustc --crate-name libc /Users/aleksandrivanov/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.42/src/lib.rs --crate-type lib --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=dce309634355ac97 -C extra-filename=-dce309634355ac97 --out-dir /Users/aleksandrivanov/Sandbox/Projects/tetris/rgame/target/x86_64-apple-ios/release/deps --target x86_64-apple-ios -L dependency=/Users/aleksandrivanov/Sandbox/Projects/tetris/rgame/target/x86_64-apple-ios/release/deps -L dependency=/Users/aleksandrivanov/Sandbox/Projects/tetris/rgame/target/release/deps --cap-lints allow -Zembed-bitcode`
...
``

[rex-remind101]

Any updates here? Heavily considering a shared core library for our app and without this fixed c++ is the likely option.

[volodg]

@chrisballinger, Can it be related to https://github.com/rust-lang/rust/issues/52686?