Add Host MacOS support

[mikee47]

This PR adds support for the Host Emulator to MacOS.

See #2773.

64-bit only

Standard compiler is 'Apple Clang', supports only 64-bit builds: no 32-bit compatibility. GCC can be installed via brew install gcc and compiles as usual with the -m32 flag. The GNU linker is absent from brew's coreutils package (see https://github.com/Homebrew/homebrew-core/issues/17794) so the standard system linker must be used. This only supports 64-bit targets. Apple have removed 32-bit application support from their ecosystem.

Advantages No need for 32-bit compatibibility libraries in GNU/Linux. For MacOS build and run host emulator using standard Apple tools, no GCC required. General improvements in portability, less tolerant of incorrect type usage.

Disadvantages Types such as long, size_t now 64-bit, don't match target hardware Pointers are 64-bit so cannot be cast to 32-bit intrinsics. Code must use correct intptr_t types. Must use ENABLE_STORAGE_SIZE64=1 as emulated flash addresses must be 64-bit.

Symbol wrapping

MacOS linker doesn't support symbol wrapping which is used by malloc_count Component to hook malloc, etc. Don't currently have a solution for this so malloc_count is disabled for MacOS.

CI builds

Added MacOS builds plus running tests. For esp32 only IDF 5.2 is tested.

Notes

MacOS identified in makefiles with UNAME=Darwin. In code, use #ifdef __APPLE__ when running on apple toolchains. Also requires #ifdef __aarch64__ for Apple Silicon. The rbpf library requires bpf target with clang. Not supported by apple clang - use brew clang.

I managed to get MacOS Ventura running in Qemu via https://github.com/kholia/OSX-KVM. This runs in x86_64 mode, adequate for dev/testing. The github actions runner macos-latest uses arm64, so introduces a few extra quirks.

Tools

• Compiler. Standard compiler is 'Apple Clang', supports only 64-bit builds: no 32-bit compatibility. Doesn't support alias attribute so use linker to do that instead.

• Archive. Standard ar does not support

• Linker. GNU Linker not available. MacOS linker does not support symbol wrapping.

• GDB. Can be installed with brew but requires some additional steps for security reasons. Separate lldb target added.

• sed. GNU 'sed' tool has extensions which MacOS version does not support. Added SED build variable which should always point to GNU version gsed, but only for Darwin.

TODO:

• [x] Pull out fixes into separate PR, merge first
• [x] Make GNU/Linux Host build type selectable between 32/64-bit
• [x] Is there a solution to symbol wrapping for MacOS linker? Nope.
• [x] Fix interrupt simulation code - requires rewrite of signal layer
• [x] Add lldb support to framework
• [x] Fix/check macos installer
• [x] Reinstate full CI builds
• [x] How to create virtual TAP for MacOS? Document.
• [x] Review MacOS documentation

[slaff]

How to create virtual TAP for MacOS? Document.

Take a look here: https://groups.google.com/g/tunnelblick-discuss/c/v5wnQCRZ8HY/m/Q8nhFBx1BgAJ

[mikee47]

The networking works as follows. First, install tap extension:

sudo cp -r "$SMING_HOME/Arch/Host/Tools/macos/tap.kext" /Library/Extensions
sudo kextload /Library/Extensions/tap.kext

There's some security guff to manually deal with, then a reboot is required. (NB. this doesn't work in CI environment but gracefully fails.)

The Sming application must be run as sudo also. Changing permissions for /dev/tap0 doesn't help, still get network write failures. I've tested this with the HttpServer_ConfigNetwork sample, running

sudo out/Host/debug/firmware/app

Then browsing to 192.168.13.10.

This kernel extension creates The /dev/tap0 device. However, the actual tap0 interface is created dynamically when the device is opened, as can be illustrated below. This post was most helpful https://stackoverflow.com/questions/87442/virtual-network-interface-in-mac-os-x/31259641#31259641.

sudo su
exec 4<>/dev/tap0
ifconfig

So two questions remain:

1. How to do this without requiring sudo? Slaff's post above suggests that the tunnelblick daemon does this sort of stuff, but that adds way more complexity than I'd like so haven't spend time on that.
2. How to configure routing so we can access internet?

Ref. https://github.com/ntop/n2n/issues/773

[mikee47]

@slaff There's still a couple of APPVEYOR references in Tools/ci/install.sh and Sming/Arch/Host/Tools/install.sh. The latter can go but the first one relates to deployment, which is your area... separate PR?

[slaff]

The latter can go but the first one relates to deployment, which is your area... separate PR?

I will take care of this next week in a separate PR.