Building the RPCS3 emulator with LTO "forced on" breaks on the link step

[KyunLFA]

Admittedly this is a corner case as RPCS3 explicitly disallows LTO in their CMakeFiles as an old "bugfix" for a previous case where Qt apps would not work properly in runtime when LTO'd, so I'd understand if this is deemed out of scope. That said, by bypassing that check and trying to build with LTO anyway, Clang+LLD fail at the link step with any linker and both Thin and Full LTO, yet GCC passes and produces a working binary, which is the sole reason I decided to create this issue (had GCC also failed, I would assume it'd simply be a RPCS3 bug). Clang+LLD work fine when LTO is set to off.

With ThinLTO, lld (below it says "ld" but I've symlinked ld.lld to /usr/bin/ld) produces various link errors relating to Qt (that I assume is a mere coincidence) that aren't present compiling a similar project (also an emulator, C++ and Qt), the PS2 emulator PCSX2. Here are a few of the link errors, which are far too many to paste here:

ld: error: cannot preempt symbol: typeinfo for QCoreApplication
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(typeinfo for headless_application)

ld: error: cannot preempt symbol: QObject::metaObject() const
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by basic_keyboard_handler.cpp
>>>               lto.tmp:(vtable for basic_keyboard_handler)

ld: error: cannot preempt symbol: QCoreApplication::event(QEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(vtable for headless_application)

ld: error: cannot preempt symbol: QObject::eventFilter(QObject*, QEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(vtable for headless_application)

ld: error: cannot preempt symbol: QObject::timerEvent(QTimerEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(vtable for headless_application)

ld: error: cannot preempt symbol: QObject::childEvent(QChildEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(vtable for headless_application)

ld: error: cannot preempt symbol: QObject::customEvent(QEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(vtable for headless_application)

ld: error: cannot preempt symbol: QObject::connectNotify(QMetaMethod const&)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(vtable for headless_application)

ld: error: cannot preempt symbol: QObject::disconnectNotify(QMetaMethod const&)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(vtable for headless_application)

ld: error: cannot preempt symbol: QCoreApplication::notify(QObject*, QEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(vtable for headless_application)

ld: error: cannot preempt symbol: QCoreApplication::compressEvent(QEvent*, QObject*, QPostEventList*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(vtable for headless_application)

ld: error: cannot preempt symbol: QCoreApplication::staticMetaObject
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by mocs_compilation.cpp
>>>               lto.tmp:(headless_application::staticMetaObject)

ld: error: cannot preempt symbol: QObject::qt_metacast(char const*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by basic_keyboard_handler.cpp
>>>               lto.tmp:(vtable for basic_keyboard_handler)

ld: error: cannot preempt symbol: QObject::qt_metacall(QMetaObject::Call, int, void**)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by basic_keyboard_handler.cpp
>>>               lto.tmp:(vtable for basic_keyboard_handler)

ld: error: cannot preempt symbol: QObject::event(QEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by basic_keyboard_handler.cpp
>>>               lto.tmp:(vtable for basic_keyboard_handler)

ld: error: cannot preempt symbol: QObject::timerEvent(QTimerEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by basic_keyboard_handler.cpp
>>>               lto.tmp:(vtable for basic_keyboard_handler)

ld: error: cannot preempt symbol: QObject::childEvent(QChildEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by basic_keyboard_handler.cpp
>>>               lto.tmp:(vtable for basic_keyboard_handler)

ld: error: cannot preempt symbol: QObject::customEvent(QEvent*)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by basic_keyboard_handler.cpp
>>>               lto.tmp:(vtable for basic_keyboard_handler)

ld: error: cannot preempt symbol: QObject::connectNotify(QMetaMethod const&)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by basic_keyboard_handler.cpp
>>>               lto.tmp:(vtable for basic_keyboard_handler)

ld: error: cannot preempt symbol: QObject::disconnectNotify(QMetaMethod const&)
>>> defined in /usr/lib/libQt6Core.so.6.7.1
>>> referenced by basic_keyboard_handler.cpp
>>>               lto.tmp:(vtable for basic_keyboard_handler)

ld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)

I've also tried downgrading to Qt 6.6.3 which RPCS3 is intended to be used with as of now, with no change in this behavior.

With FullLTO, any linker will keep using RAM indefinitely. I know because I've built bigger programs that need a lot of RAM to compile, yet RPCS3 which is not that large comparatively, will keep wasting the entirety of my RAM, a whopping 192GB that is 96GB main memory + 96GB zRAM. Again, this behavior does not appear when using GCC instead of Clang.

Tested when building RPCS3 from git source.

To test it yourself, patch into the RPCS3 tree:

--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -116,20 +116,6 @@

 add_subdirectory(3rdparty)

-if (DISABLE_LTO)
-    if (CMAKE_C_FLAGS)
-        string(REGEX REPLACE "-flto[^ ]*" "" CMAKE_C_FLAGS ${CMAKE_C_FLAGS})
-    endif()
-    if (CMAKE_CXX_FLAGS)
-        string(REGEX REPLACE "-flto[^ ]*" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
-    endif()
-endif()
-
-string(FIND "${CMAKE_C_FLAGS} ${CMAKE_CXX_FLAGS}" "-flto" FOUND_LTO)
-if (NOT FOUND_LTO EQUAL -1)
-    message(FATAL_ERROR "RPCS3 doesn't support building with LTO, use -DDISABLE_LTO=TRUE to force-disable it")
-endif()
-
 if(NOT WIN32)
     add_compile_options(-pthread)
 endif()

If this turns out to be a bug solely on the RPCS3 side, I apologize in advance for wasting time.

LLVM versions tested: 19.0.0git (e83f5c15003516d5e1e677890ddbbd44be8151f), 18.1.6, 17.0.6, no change in observed behavior GCC version used (verified that this works with LTO): 14.1.1 20240522 Binutils version: 2.42.0 CFLAGS & CXXFLAGS: "-O3 -flto -march=znver4" LDFLAGS: unset, except when specifying the linker with -fuse-ld Host: AMD64 (Ryzen 9 7900X) Distribution: CachyOS

[KyunLFA]

I apologize, but I will not be able to interact with the issue for a few hours as I will be away and offline for a while. I anticipate that I'll be able to come back in 6-10 hours time and promptly answer any inquiries. Cheers and thanks for considering my issue.

[KyunLFA]

I can reply now, FYI

[MaskRay]

You can relink the program with -Wl,--demangle to get the mangled name, then relink with -Wl,-y,XXX.

The symbols in /usr/lib/libQt6Core.so.6.7.1 likely have the STV_PROTECTED visibility. It's invalid for an executable link that uses the symbol not to define the symbol (canonical PLT entry).

[KyunLFA]

Weird, I'm passing -Wl,--demangle to LDFLAGS (also tried passing it to C[XX]FLAGS instead) yet it doesn't produce any different output than when its absent... Other flags get passed fine, any idea on why that may be happening?

[KyunLFA]

I managed to build RPCS3 with LLVM LTO just now: By recompiling qt6-base and qt6-multimedia with -fno-PIC -fno-PIE...

Edit: FullLTO still keeps maxing out RAM however...

Second edit: The binary produced by LLVM LTO'ing RPCS3 is not functional (tested with 19.0.0git specified above and 18.1.6) , I can send the debug stacktrace if needed. I've checked once again and GCC LTO'ing RPCS3 does indeed work, even with nopic nopie Qt libraries.

[KyunLFA]

FullLTO no longer maxes out my RAM on git https://github.com/llvm/llvm-project/commit/1d4e857acdd9 , compiles correctly (with custom compiled qt6-base and qt6-multimedia), and produces a viable executable on RPCS3 https://github.com/RPCS3/rpcs3/commit/5b973448bf. Closing.