Build all tools natively for Apple Silicon Macs

NikolajSchlej commented 3 years ago

As expected, our current build of Qt-5.6.3-darwin used by unixbuild.sh is not yet compatible with arm64 used on AS Macs, needs to be updated. Ideally we should build UEFITool.app as a universal binary for macOS.

NikolajSchlej commented 3 years ago

Done, now need to upload that Qt-5.6.3-static-darwin-arm64, and update unixbuild.sh to detect the architecture and select the correct library.

NikolajSchlej commented 2 years ago

There's now a funny bug in awk of all things that Apple introduced somewhere in macOS 12.x, trying to add multibyte character support to the tool that clearly shows its age.

The result of this bug is that configure script of Qt5.6.3 (the last one on LGPLv2.1 that we want to remain on) can not determine the correct architecture, despite the actual test in 'qtbase/config.tests/arch' correctly doing it. Commandline arguments -arch and -hostarch are also already deprecated in that Qt release, out of the maintainers infinite wisdom, so we will have to patch the library sources itself (also providing the patched string to comply with LGPLv2.1) to get out of that stalemate.

qtbase_configure.zip

NikolajSchlej commented 2 years ago

Our next stop is the lack of symlinks for GCC/clang-specific ar: make[3]: gcc-ar: No such file or directory for the default -platform macx-g++, or make[3]: llvm-ar: No such file or directory for a better, but less default -platform macx-clang.

A trivial change in a relevant mkspec (removal of gcc- or llvm-) does the trick.

qtbase_mkspecs_common_clang.conf.zip

NikolajSchlej commented 2 years ago

Next in our journey to get a code written in 2017 to compile by a still-in-beta clang from Xcode 13 (I'm using the newest compiler version available in a vane hope for more of this patches to not be required the next year) is this: fontdatabases/mac/qfontengine_coretext.mm:767:20: error: qualified reference to 'QFixed' is a constructor name rather than a type in this context return QFixed::QFixed(int(CTFontGetUnitsPerEm(ctfont)));

The patch is once again trivial, just use QFixed(... instead if QFixed::QFixed(....

qtbase_src_platformsupport_fontdatabases_mac_qfontengine_coretext.mm.zip

NikolajSchlej commented 2 years ago

Yet another set of errors, now with some Apple flavour:

qcocoahelpers.mm:552:39: error: use of undeclared identifier 'InvalidContext'
    require_action(inContext != NULL, InvalidContext, err = paramErr);
                                      ^
qcocoahelpers.mm:553:38: error: use of undeclared identifier 'InvalidBounds'
    require_action(inBounds != NULL, InvalidBounds, err = paramErr);
                                     ^
qcocoahelpers.mm:554:37: error: use of undeclared identifier 'InvalidImage'
    require_action(inImage != NULL, InvalidImage, err = paramErr);

require_action is a neat Apple-originated macro for better error handling in C-based languages that is aimed at making the code more readable (as the error handling is on a single line instead of the usual bunch) and preventing the goto failure; issue (because there are no naked gotos anymore).

It's likely that an older macOS SDK exposed the require_* and alike macros directly, but the modern one doesn't, so the compiler gets really confused here.

As a solution for this specific file, knowing that the error handling is just

InvalidImage:
InvalidBounds:
InvalidContext:
        return err;

the easiest way to fix it would be to remove those macros entirely and replace them with if (ptr == NULL) return paramErr; and remove the now-unused goto labels.

qtbase_src_plugins_platforms_cocoa_qcocoahelpers.mm.zip

NikolajSchlej commented 2 years ago

Success, and the final configure command line ended up like this: ./configure -prefix /opt/qt56sm -nomake examples -nomake tests -skip qt3d -skip qtactiveqt -skip qtcanvas3d -skip qtconnectivity -skip qtdeclarative -skip qtdoc -skip qtgraphicaleffects -skip qtlocation -skip qtmultimedia -skip qtquickcontrols -skip qtquickcontrols2 -skip qtscript -skip qtsensors -skip qtserialbus -skip qtserialport -skip qtsvg -skip qtwebchannel -skip qtwebengine -skip qtwebsockets -skip qtwebview -skip qtxmlpatterns -skip qtwayland -skip qttools -no-dbus -no-qml-debug -no-cups -no-opengl -no-iconv -no-dbus -no-icu -no-fontconfig -no-freetype -qt-harfbuzz -no-compile-examples -no-gif -no-libpng -no-libjpeg -no-debug-and-release -static -opensource -confirm-license -reduce-exports -release -ltcg -platform macx-clang

NikolajSchlej commented 2 years ago

If anybody is interested in testing the resulting native arm64 binaries, here they are.

UEFIExtract_NE_A60_mac_arm64.zip UEFIFind_NE_A60_mac_arm64.zip UEFITool_NE_A60_mac_arm64.zip

NikolajSchlej commented 2 years ago

What remains to consider this issue resolved:

make the modified files uploaded above into a single qt563-macx-clang-arm64.patch file
upload the arm64 library files to GitHub for CI/CD scripts to use (as recompiling the library itself every time we need to build anything from this repo makes no sense)
include architecture detection into unixbuild.sh to select and download a proper set of library files
setup a CI/CD runner on an Apple Silicon Mac
make release builds of all tools into universal apps (using lipo)

Support for arm64 in Linux is out of scope for this issue, might need another one later on.

vit9696 commented 2 years ago

For building this on x86 one would also need to fix qsimd_p.h, which incorrectly detects _bit_scan_reverse/_bit_scan_forward intrinsic presence:

qtbase_src_corelib_tools_qsimd_p.h.h.zip

vit9696 commented 2 years ago

Instead of making separate Qt builds I rather changed Qt macx-clang spec to produce fat binaries (arm64/x86_64) as it should normally do. Manual hacking in configure was required to make it bundle sse2 code.

Latest patch: qt-everywhere-opensource-src-5.6.3.patch.zip

Up-to-date configure script should be as follows, no-pch is needed to build fat binaries:

./configure -prefix /opt/qt56sm -nomake examples -nomake tests -skip qt3d -skip qtactiveqt -skip qtcanvas3d -skip qtconnectivity -skip qtdeclarative -skip qtdoc -skip qtgraphicaleffects -skip qtlocation -skip qtmultimedia -skip qtquickcontrols -skip qtquickcontrols2 -skip qtscript -skip qtsensors -skip qtserialbus -skip qtserialport -skip qtsvg -skip qtwebchannel -skip qtwebengine -skip qtwebsockets -skip qtwebview -skip qtxmlpatterns -skip qtwayland -skip qttools -no-dbus -no-qml-debug -no-cups -no-opengl -no-iconv -no-dbus -no-icu -no-fontconfig -no-freetype -qt-harfbuzz -no-compile-examples -no-gif -no-libpng -no-libjpeg -no-debug-and-release -static -opensource -confirm-license -reduce-exports -release -ltcg -platform macx-clang -no-pch

vit9696 commented 2 years ago

@NikolajSchlej, new_engine branch ci generates fat binaries from now on. Please test them on new macs and close the issue.

NikolajSchlej commented 2 years ago

Tested already, worked as expected for me on M1 MBP.

LongSoft / UEFITool

Build all tools natively for Apple Silicon Macs #228