Broken build macOS 11.1 Apple M1 chip

[eizyaev]

Hello, It's my first time trying to build SerenityOS (So probably I am missing something obvious here), I am getting the following error:

Did anyone stumbled upon this by any chance and know how to solve it?

$ cd .. $ cd Build $ cmake .. -DCMAKE_C_COMPILER=gcc-10 -DCMAKE_CXX_COMPILER=g++-10 $ make

... Scanning dependencies of target TestBinarySearch [ 6%] Building CXX object AK/Tests/CMakeFiles/TestBinarySearch.dir/TestBinarySearch.cpp.o [ 6%] Linking CXX executable TestBinarySearch [ 6%] Built target TestBinarySearch Scanning dependencies of target Regex [ 6%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Regex.dir/Regex.cpp.o [ 6%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Regex.dir//C/Regex.cpp.o [ 6%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Regex.dir//RegexByteCode.cpp.o [ 6%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Regex.dir//RegexLexer.cpp.o [ 6%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Regex.dir//RegexMatcher.cpp.o [ 6%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Regex.dir//RegexParser.cpp.o [ 6%] Linking CXX executable Regex [ 6%] Built target Regex Scanning dependencies of target RegexLibC [ 6%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/RegexLibC.dir/RegexLibC.cpp.o [ 6%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/RegexLibC.dir//C/Regex.cpp.o [ 7%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/RegexLibC.dir//RegexByteCode.cpp.o [ 7%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/RegexLibC.dir//RegexLexer.cpp.o [ 7%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/RegexLibC.dir//RegexMatcher.cpp.o [ 7%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/RegexLibC.dir//RegexParser.cpp.o [ 7%] Linking CXX executable RegexLibC [ 7%] Built target RegexLibC Scanning dependencies of target Benchmark [ 7%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Benchmark.dir/Benchmark.cpp.o [ 7%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Benchmark.dir//C/Regex.cpp.o [ 7%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Benchmark.dir//RegexByteCode.cpp.o [ 8%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Benchmark.dir//RegexLexer.cpp.o [ 8%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Benchmark.dir//RegexMatcher.cpp.o [ 8%] Building CXX object Libraries/LibRegex/Tests/CMakeFiles/Benchmark.dir/__/RegexParser.cpp.o [ 8%] Linking CXX executable Benchmark [ 8%] Built target Benchmark Scanning dependencies of target boot [ 8%] Building ASM object Kernel/CMakeFiles/boot.dir/Arch/i386/Boot/boot.S.o i686-pc-serenity-gcc: error: x86_64: No such file or directory i686-pc-serenity-gcc: error: unrecognized command-line option '-arch' make[2]: [Kernel/CMakeFiles/boot.dir/Arch/i386/Boot/boot.S.o] Error 1 make[1]: [Kernel/CMakeFiles/boot.dir/all] Error 2 make: *** [all] Error 2

[ADKaster]

Well, obvious question, you built the cross-compiler in Toolchain/ first? There's likely some missing option to the gcc configure lines we're using in BuildIt.sh to properly create a cross-compiler from darwin-aarch64 to i686-serenity.

[ADKaster]

As an alternative, if there's apple silicon ports of docker and/or VirtualBox, you could try using a linux docker container / VM to build? Kind of uncharted territory here :D

[eizyaev]

@ADKaster thanks for replying. As I am both new to macOS environment (+the new apple silicon in this machine) and to this project, for the meantime I will try to build it on my Linux machine and probably sometime later I will return to try configuring the cross-compiler properly when building the Toolchain.

[willmcpherson2]

OK, I've got Serenity running in QEMU on my M1 Macbook Air. These instructions are a starting point, but there may be unnecessary steps and there is definitely room for improvement in terms of convenience and reliability. I really just wanted to get a proof-of-concept. So if you're running an M1 machine with Big Sur 11.2, please try this out and let me know if you get stuck.

Replace Serenity's Toolchain/BuildIt.sh with this new one.

Install Xcode from the App Store.

You may also need to open it and agree to the license.

You may also need to install Command Line Tools for your version for Xcode.

You'll need homebrew. Check out their website for how to install it.

Install these dependencies:

brew install coreutils e2fsprogs m4 autoconf libtool automake bash gcc@10 ninja
brew install --cask osxfuse

e2fsprogs might not be in your $PATH. If that's the case, tell homebrew to link it, then update your $PATH:

brew ln --force e2fsprogs
echo 'export PATH="/opt/homebrew/opt/e2fsprogs/bin:$PATH"' >> ~/.zshrc
echo 'export PATH="/opt/homebrew/opt/e2fsprogs/sbin:$PATH"' >> ~/.zshrc

Assuming you've downloaded the Serenity source, run the provided script to install FUSE:

serenity/Toolchain/BuildFuseExt2.sh

FUSE may require updating to actually use it. This can be done from System Preferences:

System Preferenes -> FUSE -> Update

You will need to build QEMU from source. Put it wherever you want. I'm going to put it on my Desktop:

cd ~/Desktop
git clone https://git.qemu.org/git/qemu.git
cd qemu
mkdir build
cd build
../configure --target-list=i386-softmmu --enable-cocoa --enable-debug-tcg
make -j$(nproc)

You will then need to update your $PATH:

echo 'export PATH="~/Desktop/qemu/build:$PATH"' >> ~/.zshrc

Now we build the toolchain:

cd serenity/Toolchain
./BuildIt.sh

Finally, we can build Serenity:

cd serenity/Build
cmake .. -G Ninja -DCMAKE_C_COMPILER=gcc-10 -DCMAKE_CXX_COMPILER=g++-10 -DCMAKE_SYSTEM_NAME=Darwin
ninja install
ninja image
ninja run

[thomas-mangin]

Feedback on the instructions above:

Installing xcode does require accepting the licence.

Compiling qemu required the following addition:

> brew install pkg-config
> brew install glib
> brew install pixman

serenity also requires cmake

> brew install cmake

A few reboots were required to get Fuse working (including one to allow kernel extension to run on the system, which requires to go to the recovery mode, and only worked with 11.2. I failed a few times to enable the kernel extension before deciding to upgrade os x)

XCode tools installed from the developer site but still required, as otherwise ninja image would fail due to xcrun

> xcode-select --install

ninja image reports a failure umounting filesystem... rmdir: mnt: Directory not empty but the disk image created works as expected when started with ninja run.

[willmcpherson2]

@thomas-mangin

Thanks for the feedback.

1. So cmake and ninja must both be explicitly installed?
2. Did you need to update FUSE from System Preferences?
3. I also had to reboot and enter recovery mode, but I don't remember when. Is it after installing/updating FUSE? Or only once you tried to use it?
4. Do we need to manually download Command Line Tools from the Apple website or does xcode-select --install achieve that?
5. I also got umounting filesystem... rmdir: mnt: Directory not empty, but only initially. In subsequent builds that error message didn't show up. Either way we can just change rmdir to rm -rf in that script.

[willmcpherson2]

There are quite a few issues with these instructions that we should try to iron out:

1. We're disabling PCH in our GCC build (see BuildIt.sh:176). I really have no idea what the consequences are. Check out this issue over at riscv/homebrew-riscv. If it is OK, we'll do it conditionally based on the host architecture.

2. We're using --enable-debug-tcg to build QEMU... I have no idea why, but I get this error otherwise:

qemu-system-i386: qemu_mprotect__osdep: mprotect failed: Permission denied
**
ERROR:../tcg/tcg.c:734:tcg_region_init: assertion failed: (!rc)
Bail out! ERROR:../tcg/tcg.c:734:tcg_region_init: assertion failed: (!rc)

This call to mprotect is failing.

If anyone's up for some investigating, try grepping through the QEMU code for CONFIG_DEBUG_TCG, as that's what the --enable-debug-tcg flag defines.

See this helpful blogpost for more details on QEMU for M1.

Obviously QEMU support for M1 is bleeding edge, but things are looking good. I think if we wait a few weeks, brew install qemu will be sufficient.

3. "Go build QEMU somewhere" is annoying. Serenity already has a method for building and running QEMU from source, so we should just use that. We should be able to put this into a script and avoid the export in .zshrc. Speaking of which, we might be able to avoid exporting the e2fsprogs paths by using symlinks. Best to avoid touching .zshrc entirely.

[thomas-mangin]

@willmcpherson2

To answer your questions:

1. Yes, both were explicitly required.
2. I did, but I can not say if it could have worked without, and as you need to reboot anyway to enable the kernel extension
3. I can not say if you need to update FUSE but I did.
4. I also had to reboot and enter recovery mode, but I don't remember when. Is it after installing/updating FUSE? Or only once you tried to use it? It became clear your need to reboot to use fuse when you try to make the image, before that FUSE seems to be fully installed but is not. I assume using the commands attempt to use the kernel feature, which then dynamically attempt to load the module.
5. I installed the dmg from Apple dev site, but it was not enough to get the code compiling so I had to use xcode-select. I can not say if xcode-select on its own would be enough (but I suspect it may). I do not know if xcode-select comes with xcode or the xcode command-line tools.

I also got umounting filesystem... rmdir: mnt: Directory not empty, but only initially. In subsequent builds that error message didn't show up. Either way, we can just change rmdir to rm -rf in that script. :+1:

[IvanHansgaardHansen]

@willmcpherson2 I just want to comment that I was able to get it working using your instuctions - thanks!

It seems to be running super slow, though. As a reference loadtimes for the default page in the browser is often in the 1000s of milliseconds, not sub-100 as on my Linux-machine with ~1/5 of the CPU horsepower.

[willmcpherson2]

@IvanHansgaardHansen

Great! It is definitely a bit slow. Or rather, KVM is fast. We might be able to use HVF?

I think emulating Serenity on aarch64 will just always be a bit slower.

[willmcpherson2]

Update

The qemu build is pretty normal now:

git clone https://git.qemu.org/git/qemu.git
cd qemu
mkdir build
cd build
../configure --target-list=i386-softmmu
make -j$(nproc)

Can anyone else confirm?

Also wrt slowness, does anyone else get a short system-wide pause every ~500ms? If I drag my mouse across the screen, it will stutter at consistent intervals. If you open Piano and hold down a note, the note will cut out and the bar will stop scrolling across the screen, at the same time. This issue makes Piano basically untestable for me 😢

[willmcpherson2]

Update

1.

Seems that qemu v6.0.0-rc4 works and we can even compile with hvf:

git clone https://git.qemu.org/git/qemu.git
cd qemu
git checkout v6.0.0-rc4
mkdir build
cd build
../configure --target-list=i386-softmmu --enable-hvf
make -j$(nproc)

Although I need to figure out how to actually get qemu to run with hvf...

2.

Here's a more up-to-date BuildIt.sh

3.

Once you've built the toolchain, these are the build commands:

cd ../Build/i686
cmake ../.. -G Ninja -DCMAKE_C_COMPILER=gcc-10 -DCMAKE_CXX_COMPILER=g++-10 -DCMAKE_SYSTEM_NAME=Darwin
ninja install && ninja image && ninja run

[ADKaster]

Is this issue understood well enough now that it can be promoted to a section in the docs?

[willmcpherson2]

Ok, for upstreaming:

1.

Are our prerequisites different from Intel macOS?

2.

For some reason we require the additional cmake flag -DCMAKE_SYSTEM_NAME=Darwin. Can we simply add this or will it break Intel macOS?

3.

Should we just put our BuildIt.sh patch under a uname conditional? Do Intel builds even use PCH? What happens when we upgrade to gcc 11?

4.

Looks like we've updated BuildQemu.sh to use qemu-6.0.0-rc3 and soon qemu-6.0.0-rc4. I just ran the current one and it works for me, so that's sorted.

[gunnarbeutner]

GCC 11 still fails to build without applying your patch (haven't tested with the patch or with arch -x86_64 though):

[gcc+libgcc/install] g++ -std=gnu++11 -no-pie   -g0 -O2 -DIN_GCC  -DCROSS_DIRECTORY_STRUCTURE  -fno-strict-aliasing -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings   -DHAVE_CONFIG_H  -o cc1 c/c-lang.o c-family/stub-objc.o attribs.o c/c-errors.o c/c-decl.o c/c-typeck.o c/c-convert.o c/c-aux-info.o c/c-objc-common.o c/c-parser.o c/c-fold.o c/gimple-parser.o c-family/c-common.o c-family/c-cppbuiltin.o c-family/c-dump.o c-family/c-format.o c-family/c-gimplify.o c-family/c-indentation.o c-family/c-lex.o c-family/c-omp.o c-family/c-opts.o c-family/c-pch.o c-family/c-ppoutput.o c-family/c-pragma.o c-family/c-pretty-print.o c-family/c-semantics.o c-family/c-ada-spec.o c-family/c-ubsan.o c-family/known-headers.o c-family/c-attribs.o c-family/c-warn.o c-family/c-spellcheck.o i386-c.o default-c.o \
[gcc+libgcc/install]      cc1-checksum.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a libcommon.a ../libcpp/libcpp.a  -liconv ../libbacktrace/.libs/libbacktrace.a ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a  -L/Users/gunnar/serenity/Toolchain/Build/i686/gcc/./isl/.libs  -lisl -L/Users/gunnar/serenity/Toolchain/Build/i686/gcc/./gmp/.libs -L/Users/gunnar/serenity/Toolchain/Build/i686/gcc/./mpfr/src/.libs -L/Users/gunnar/serenity/Toolchain/Build/i686/gcc/./mpc/src/.libs -lmpc -lmpfr -lgmp   -L./../zlib -lz
[gcc+libgcc/install] clang: warning: argument unused during compilation: '-no-pie' [-Wunused-command-line-argument]
[gcc+libgcc/install] Undefined symbols for architecture arm64:
[gcc+libgcc/install]   "_host_hooks", referenced from:
[gcc+libgcc/install]       c_common_no_more_pch() in c-pch.o
[gcc+libgcc/install]       toplev::main(int, char**) in libbackend.a(toplev.o)
[gcc+libgcc/install]       gt_pch_save(__sFILE*) in libbackend.a(ggc-common.o)
[gcc+libgcc/install]       gt_pch_restore(__sFILE*) in libbackend.a(ggc-common.o)
[gcc+libgcc/install] ld: symbol(s) not found for architecture arm64
[gcc+libgcc/install] clang: error: linker command failed with exit code 1 (use -v to see invocation)
[gcc+libgcc/install] make[1]: *** [cc1] Error 1
[gcc+libgcc/install] make: *** [install-gcc] Error 2

Personally I'd rather have the patch in Toolchain/Patches/gcc.patch and Ports/gcc/patches/gcc.patch - assuming it builds without issues on Intel Macs. This way updating the toolchain is easier.

[willmcpherson2]

I think the latest Xcode has done something bad to the qemu build. I kept getting this same error:

[852/2297] Compiling C++ object libcommon.fa.p/disas_libvixl_vixl_utils.cc.o
FAILED: libcommon.fa.p/disas_libvixl_vixl_utils.cc.o 
c++ -Ilibcommon.fa.p -I. -I.. -I../capstone/include/capstone -I../dtc/libfdt -I../slirp -I../slirp/src -Iqapi -Itrace -Iui -Iui/shader -I/opt/homebrew/Cellar/pixman/0.40.0/include/pixman-1 -I/opt/homebrew/Cellar/libpng/1.6.37/include/libpng16 -I/opt/homebrew/Cellar/jpeg/9d/include -I/opt/homebrew/Cellar/gnutls/3.6.15/include -I/opt/homebrew/Cellar/nettle/3.7.2/include -I/opt/homebrew/Cellar/libtasn1/4.16.0_1/include -I/opt/homebrew/Cellar/libidn2/2.3.0/include -I/opt/homebrew/Cellar/p11-kit/0.23.22/include/p11-kit-1 -I/opt/homebrew/Cellar/libffi/3.3_3/include -I/opt/homebrew/Cellar/glib/2.68.1/include -I/opt/homebrew/Cellar/glib/2.68.1/include/glib-2.0 -I/opt/homebrew/Cellar/glib/2.68.1/lib/glib-2.0/include -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/Cellar/pcre/8.44/include -I/opt/homebrew/Cellar/glib/2.68.1/include/gio-unix-2.0 -I/opt/homebrew/Cellar/libusb/1.0.24/include/libusb-1.0 -fcolor-diagnostics -Wall -Winvalid-pch -Wnon-virtual-dtor -std=gnu++11 -O2 -g -iquote . -iquote /Users/will/Desktop/qemu -iquote /Users/will/Desktop/qemu/include -iquote /Users/will/Desktop/qemu/disas/libvixl -iquote /Users/will/Desktop/qemu/tcg/aarch64 -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DOS_OBJECT_USE_OBJC=0 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wundef -Wwrite-strings -fno-strict-aliasing -fno-common -fwrapv -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wendif-labels -Wexpansion-to-defined -Wno-initializer-overrides -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-string-plus-int -Wno-typedef-redefinition -Wno-tautological-type-limit-compare -Wno-psabi -fstack-protector-strong -DSTRUCT_IOVEC_DEFINED -MD -MQ libcommon.fa.p/disas_libvixl_vixl_utils.cc.o -MF libcommon.fa.p/disas_libvixl_vixl_utils.cc.o.d -o libcommon.fa.p/disas_libvixl_vixl_utils.cc.o -c ../disas/libvixl/vixl/utils.cc
In file included from ../disas/libvixl/vixl/utils.cc:27:
In file included from /Users/will/Desktop/qemu/disas/libvixl/vixl/utils.h:31:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/cmath:308:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/math.h:309:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/type_traits:417:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/cstddef:37:
../version:1:1: error: expected unqualified-id
6.0.50
^

The standard library is trying to include the qemu VERSION file!?

Fortunately I found this gist, this comment and this patch. That patch actually doesn't work, but I managed to fix it.

So here's the patch. It works on d90f154867ec0ec22fd719164b88716e8fd48672.