Unable to use ThinLTO

[DoDoENT]

I'm experimenting with ThinLTO instead of monolithic LTO, but unfortunately it always fails.

I get one of the two following errors at link time:

wasm-ld: error: inconsistent LTO Unit splitting (recompile with -fsplit-lto-unit)

or

wasm-ld: error: Expected at most one ThinLTO module per bitcode file

I've tried recompiling with -fsplit-lto-unit, as suggested, but I still get the same error. Based on this, LTO unit splitting should be enabled by default as soon as I enable -fwhole-program-vtables, which I always have enabled.

Is ThinLTO supported by the emscripten?

[DoDoENT]

OK, now I see that this should be fixed in 2.0.33

[DoDoENT]

Well, it's not fixed in 2.0.33 😞

Now I consistently get the following error:

wasm-ld: error: inconsistent LTO Unit splitting (recompile with -fsplit-lto-unit)

Note that I have added -fsplit-lto-unit to my compile flags.

[DoDoENT]

The error is gone if I compile without -fwhole-program-vtables flag.

[pirama-arumuga-nainar]

Is it possible you have stale object files that were built without -fwhole-program-vtables/-fsplit-lto-unit?

[DoDoENT]

Is it possible you have stale object files that were built without -fwhole-program-vtables/-fsplit-lto-unit?

I'm pretty confident that the answer is no because everything was built from the source, but I'll re-check that next week (it may be possible that some 3rd party packages that we've rebuilt have overridden flags that we've passed globally (notably -flto=thin -fwhole-program-vtables).

But wouldn't that case then also cause similar or even the same issue on our other LLVM-based builds (Android NDK, MacOS/iOS, Linux)? We are passing the same flags to all our clang-based builds.

[DoDoENT]

Good news!

I have a repro.

I've managed to recycle my old reproducer for different emscripten issue to also reproduce this issue.

I hope it will help with resolving the issue.

[DoDoENT]

I've managed to reproduce the issue on desktop LLVM when statically linking libc++ that was built using LLVM_ENABLE_LTO=Thin.

In that case, the libc++ was built without -fwhole-program-vtables and that caused the same error as here.

Therefore, based on this comment, I've changed get_base_cflags from tools/system_libs.py to also include -fwhole-program-vtables and purged the cache.

This changed removed the above error, but caused following linker warnings:

|| wasm-ld: warning: function signature mismatch: _ZNKSt3__25ctypeIcE9do_narrowEPKcS3_cPc
|| >>> defined as () -> void in /Users/dodo/.conan/data/emsdk_installer/2.0.34/microblink/stable/package/743cf0321be3152777da4d05247a66d1552e70a2/.emscripten_cache/sysroot/lib/wasm32-emscripten/thinlto/libc++-noexcept.a(locale.o)
|| >>> defined as (i32, i32, i32, i32, i32) -> i32 in lto.tmp

... and similar, for different symbols.

However, the binary was there so I tried to run it in the browser, but unfortunately it does not work. It crashes with the similar error message:

Uncaught RuntimeError: null function or function signature mismatch
    at SweatShopTest.wasm:0x7907
    at Ka (SweatShopTest.js:13)
    at b (SweatShopTest.js:63)
    at SweatShopTest.js:65

The same happens if I force full LTO for system libraries. (It's possible to mix ThinLTO and FullLTO objects).

[sbc100]

Does this problem go away if you don't use LTO at all for system libraries?

[DoDoENT]

Does this problem go away if you don't use LTO at all for system libraries?

No, it doesn't 😞 .

However, I'm still working on setting up the build of another project just to rule out the case that this function signature mismatch is not just a different manifestation of this issue (I've used the reproducer of that issue to bootstrap the reproducer of this issue here).

[DoDoENT]

I've tried on a different project, and guess what?

A new ICE:

|| Assertion failed: (DataLocations.count(&WS) > 0), function writeOneObject, file /opt/s/w/ir/cache/builder/emscripten-releases/llvm-project/llvm/lib/MC/WasmObjectWriter.cpp, line 1794.
|| PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
|| Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
|| 0  libLLVM.dylib            0x000000010ad4c67b llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 43
|| 1  libLLVM.dylib            0x000000010ad4b448 llvm::sys::RunSignalHandlers() + 248
|| 2  libLLVM.dylib            0x000000010ad4cce0 SignalHandler(int) + 288
|| 3  libsystem_platform.dylib 0x00007ff807aace2d _sigtramp + 29
|| 4  libsystem_platform.dylib 000000000000000000 _sigtramp + 18446603370452300272
|| 5  libsystem_c.dylib        0x00007ff8079e3d10 abort + 123
|| 6  libsystem_c.dylib        0x00007ff8079e30be err + 0
|| 7  libLLVM.dylib            0x000000010c69070a (anonymous namespace)::WasmObjectWriter::writeOneObject(llvm::MCAssembler&, llvm::MCAsmLayout const&, (anonymous namespace)::WasmObjectWriter::DwoMode) + 34842
|| 8  libLLVM.dylib            0x000000010c686e38 (anonymous namespace)::WasmObjectWriter::writeObject(llvm::MCAssembler&, llvm::MCAsmLayout const&) + 152
|| 9  libLLVM.dylib            0x000000010c62f36c llvm::MCAssembler::Finish() + 76
|| 10 libLLVM.dylib            0x000000010b73ef33 llvm::AsmPrinter::doFinalization(llvm::Module&) + 5683
|| 11 libLLVM.dylib            0x000000010aed7156 llvm::FPPassManager::doFinalization(llvm::Module&) + 70
|| 12 libLLVM.dylib            0x000000010aecfada llvm::legacy::PassManagerImpl::run(llvm::Module&) + 4522
|| 13 libLLVM.dylib            0x000000010c5f8a3c codegen(llvm::lto::Config const&, llvm::TargetMachine*, std::__2::function<std::__2::unique_ptr<llvm::NativeObjectStream, std::__2::default_delete<llvm::NativeObjectStream> > (unsigned int)>, unsigned int, llvm::Module&, llvm::ModuleSummaryIndex const&) + 1292
|| 14 libLLVM.dylib            0x000000010c5f990a llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::__2::function<std::__2::unique_ptr<llvm::NativeObjectStream, std::__2::default_delete<llvm::NativeObjectStream> > (unsigned int)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::StringMap<std::__2::unordered_set<unsigned long long, std::__2::hash<unsigned long long>, std::__2::equal_to<unsigned long long>, std::__2::allocator<unsigned long long> >, llvm::MallocAllocator> const&, llvm::DenseMap<unsigned long long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long long>, llvm::detail::DenseMapPair<unsigned long long, llvm::GlobalValueSummary*> > const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int> >, std::__2::vector<std::__2::pair<llvm::StringRef, llvm::BitcodeModule>, std::__2::allocator<std::__2::pair<llvm::StringRef, llvm::Bitc
|| 15 libLLVM.dylib            0x000000010c5f978f llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::__2::function<std::__2::unique_ptr<llvm::NativeObjectStream, std::__2::default_delete<llvm::NativeObjectStream> > (unsigned int)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::StringMap<std::__2::unordered_set<unsigned long long, std::__2::hash<unsigned long long>, std::__2::equal_to<unsigned long long>, std::__2::allocator<unsigned long long> >, llvm::MallocAllocator> const&, llvm::DenseMap<unsigned long long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long long>, llvm::detail::DenseMapPair<unsigned long long, llvm::GlobalValueSummary*> > const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int> >, std::__2::vector<std::__2::pair<llvm::StringRef, llvm::BitcodeModule>, std::__2::allocator<std::__2::pair<llvm::StringRef, llvm::Bitc
|| 16 libLLVM.dylib            0x000000010c5f23a7 (anonymous namespace)::InProcessThinBackend::runThinLTOBackendThread(std::__2::function<std::__2::unique_ptr<llvm::NativeObjectStream, std::__2::default_delete<llvm::NativeObjectStream> > (unsigned int)>, std::__2::function<std::__2::function<std::__2::unique_ptr<llvm::NativeObjectStream, std::__2::default_delete<llvm::NativeObjectStream> > (unsigned int)> (unsigned int, llvm::StringRef)>, unsigned int, llvm::BitcodeModule, llvm::ModuleSummaryIndex&, llvm::StringMap<std::__2::unordered_set<unsigned long long, std::__2::hash<unsigned long long>, std::__2::equal_to<unsigned long long>, std::__2::allocator<unsigned long long> >, llvm::MallocAllocator> const&, llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo> > const&, std::__2::map<unsigned long long, llvm::GlobalValue::LinkageTypes, std::__2::less<unsigned long long>, std::__2::allocator<std::__2::pair<unsigned long long const, llvm::GlobalValue::LinkageTypes> > > const&, llvm::DenseMap<unsigne
|| 17 libLLVM.dylib            0x000000010c5f21e2 void std::__2::__function::__policy_invoker<void ()>::__call_impl<std::__2::__function::__default_alloc_func<std::__2::__bind<(anonymous namespace)::InProcessThinBackend::start(unsigned int, llvm::BitcodeModule, llvm::StringMap<std::__2::unordered_set<unsigned long long, std::__2::hash<unsigned long long>, std::__2::equal_to<unsigned long long>, std::__2::allocator<unsigned long long> >, llvm::MallocAllocator> const&, llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo> > const&, std::__2::map<unsigned long long, llvm::GlobalValue::LinkageTypes, std::__2::less<unsigned long long>, std::__2::allocator<std::__2::pair<unsigned long long const, llvm::GlobalValue::LinkageTypes> > > const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int> >, std::__2::vector<std::__2::pair<llvm::StringRef, llvm::BitcodeModul
|| 18 libLLVM.dylib            0x000000010ace95c6 void* llvm::thread::ThreadProxy<std::__2::tuple<llvm::ThreadPool::ThreadPool(llvm::ThreadPoolStrategy)::$_0> >(void*) + 662
|| 19 libsystem_pthread.dylib  0x00007ff807a97514 _pthread_start + 125
|| 20 libsystem_pthread.dylib  0x00007ff807a9302f thread_start + 15
|| em++: error: '/Users/dodo/.conan/data/emsdk_installer/2.0.34/microblink/stable/package/743cf0321be3152777da4d05247a66d1552e70a2/upstream/bin/wasm-ld @/var/folders/w1/67dfjvfs6sd66rd33brg4d5r0000gn/T/emscripten_qjgw5i34.rsp' failed (received SIGABRT (-6))

This case used also -fwhole-program-vtables for system libs.

[DoDoENT]

Same crash even if I put system libs into full LTO mode and without LTO. It appears that this ICE is specific to building with -fwhole-program-vtables.

[sbc100]

This crash also looks different in that it is in that wasm backend and not in the compiler so is specific to wasm.

This also only occurs with thinLTO right?

[DoDoENT]

This occurs when I build my code with ThinLTO and -fwhole-program-vtables.

If I only enable -flto=thin, but without -fwhole-program-vtables, then it works (I can build my code in that case).

[sbc100]

That makes sense I guess that there would be issues here, since we don't have any testing on thingLTO in emscripten today (and certainly none with -fwhole-program-vtables). I'm looking at enabling thinlto in our core test suite at the moment. Once that is done I'll try enabling -fwhole-program-vtables and see if it gives me a simpler reproducer for that crash.

Would you mind opening an llvm bug against the WebAssembly backend with that above ICE (the Assertion failed: (DataLocations.count(&WS) > 0 one)?

[DoDoENT]

Would you mind opening an llvm bug against the WebAssembly backend with that above ICE (the Assertion failed: (DataLocations.count(&WS) > 0 one)?

Not at all. However, unfortunately, I don't have an open-source reproducer at the moment.

[DoDoENT]

Reported here.