Support .NET on Apple Silicon with Rosetta 2 emulation

sdmaclea commented 3 years ago

Apple has announced plans to transition its Mac hardware line to a new Arm64-based chip that they refer to as “Apple Silicon”.

Initial .NET support will be through .NET running on the Rosetta 2 emulator. Longer term native support for Apple Silicon is planned for .NET 6.

While it is hoped that Rosetta 2 emulation will just work, the .NET runtime is complicated and real issues will make this a non-trivial task.

Current known issues

[x] Apple Silicon uses a 16K memory page size. The .NET 5 stack probe code doesn't handle this yet. #45226. Per Apple this only affects the DTK and is fixed on M1 Silicon. Edit: I've verified that on real M1 device, the page size is 4K and the related issue doesn't occur.
[x] Rosetta 2 emulation crashes with a fatal failure when calling with thread_get_state x86_FLOAT_STATE64. This is because the emulator does not emulate AVX support, but the function should simply return an error. Edit: This is fixed in the macOS 11.2 beta release.
[x] Rosetta 2 emulation doesn't populate exceptionState.__trapno for other kernel entry than hardware exceptions (for example for syscalls). This means we fail to inject code necessary for garbage collection and sometimes deadlock. Edit: This is at least partially fixed in the macOS 11.2 beta release, but I can still see hangs during .NET runtime / tests managed parts compilation. It is being investigated at the Apple side.
[ ] ~~With #45226 & https://github.com/janvorli/runtime/commit/aee81acd99b9c0e6a406bad3b98c278669c7cc67 19 runtime tests are failing under Rosetta 2 emulation which pass on macOS native x64~~ All the coreclr Pri 1 tests are now passing except two tests (mentioned in the comments below) that are failing with: assertion failed: GPR thread_set_state is unsupported while in sa_tramp. (ThreadContextRegisterState.cpp:1250 thread_set_state_gpr_64)
[x] Debugging using VS Code doesn't work. It was partially fixed in the macOS 11.2 beta release, now it is possible to successfully run an application under the debugger and break on a breakpoint. But attempt to single step or continue from that state still fails. It is caused by iret instruction emulation that doesn't honor the trace flag. Edit: This is fixed in macOS 11.2 beta 2
[x] New issue in macOS 11.2 beta - dotnet build and posibly other .NET applications often fail with assertion failed [abi_info.u.translated_code.instruction_extents.kind == InstructionOffsetKind::Syscall]: on sigreturn exit path but instruction isn't marked as a syscall (ThreadContextRegisterState.cpp:381 x86_gpr_state_from_arm_state) Edit: This is fixed in macOS 11.2 beta 2

sdmaclea commented 3 years ago

@richlander Based on:

the qemu: uncaught target signal 6 (Aborted) - core dumped above
the docker M1 tech preview comment However, attempts to run Intel-based containers on Apple M1 machines can crash as QEMU sometimes fails to run the container.

I would assume docker is using qemu rather than Rosetta to make this work.

sdmaclea commented 3 years ago

Is that build unsigned?

It appears to be unsigned.

dotnet is adhoc signed

$ codesign -dvvv dotnet
Executable=/Users/stmaclea/git/dotnet-sdk-5.0.103-osx-x64/dotnet
Identifier=dotnet-55554944b2da1d4ea11a33dbabc8bfe88ecd1722
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=960 flags=0x2(adhoc) hashes=22+5 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=a73826b25ee1b05734de9ca3560c399a8fccac4d
CandidateCDHashFull sha256=a73826b25ee1b05734de9ca3560c399a8fccac4df0ce0a774e547440891312ee
Hash choices=sha256
CMSDigest=a73826b25ee1b05734de9ca3560c399a8fccac4df0ce0a774e547440891312ee
CMSDigestType=2
CDHash=a73826b25ee1b05734de9ca3560c399a8fccac4d
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

libraries are unsigned.

$ find . -name \*.dylib -print -exec codesign -dvvv '{}' ';'
./host/fxr/5.0.3/libhostfxr.dylib
./host/fxr/5.0.3/libhostfxr.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libcoreclr.dylib
./shared/Microsoft.NETCore.App/5.0.3/libcoreclr.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libSystem.Native.dylib
./shared/Microsoft.NETCore.App/5.0.3/libSystem.Native.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libSystem.IO.Compression.Native.dylib
./shared/Microsoft.NETCore.App/5.0.3/libSystem.IO.Compression.Native.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libSystem.Security.Cryptography.Native.Apple.dylib
./shared/Microsoft.NETCore.App/5.0.3/libSystem.Security.Cryptography.Native.Apple.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libmscordaccore.dylib
./shared/Microsoft.NETCore.App/5.0.3/libmscordaccore.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libSystem.Net.Security.Native.dylib
./shared/Microsoft.NETCore.App/5.0.3/libSystem.Net.Security.Native.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libmscordbi.dylib
./shared/Microsoft.NETCore.App/5.0.3/libmscordbi.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libhostpolicy.dylib
./shared/Microsoft.NETCore.App/5.0.3/libhostpolicy.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libSystem.Security.Cryptography.Native.OpenSsl.dylib
./shared/Microsoft.NETCore.App/5.0.3/libSystem.Security.Cryptography.Native.OpenSsl.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libdbgshim.dylib
./shared/Microsoft.NETCore.App/5.0.3/libdbgshim.dylib: code object is not signed at all
./shared/Microsoft.NETCore.App/5.0.3/libclrjit.dylib
./shared/Microsoft.NETCore.App/5.0.3/libclrjit.dylib: code object is not signed at all
./packs/Microsoft.NETCore.App.Host.osx-x64/5.0.3/runtimes/osx-x64/native/libnethost.dylib
./packs/Microsoft.NETCore.App.Host.osx-x64/5.0.3/runtimes/osx-x64/native/libnethost.dylib: code object is not signed at all

richlander commented 3 years ago

Thanks on that. Sent mail to folks to get a resolution on the signing.

leecow commented 3 years ago

Hm, I downloaded and installed a fresh 5.0.103 from dot.net and everything checks out. Example ...

Executable=/usr/local/share/dotnet/host/fxr/5.0.3/libhostfxr.dylib
Identifier=libhostfxr
Format=Mach-O thin (x86_64)
CodeDirectory v=20500 size=3414 flags=0x10000(runtime) hashes=101+2 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=a9b0dd7d25ba9fb90f955011ffb78e5133c545e1
CandidateCDHashFull sha256=a9b0dd7d25ba9fb90f955011ffb78e5133c545e1d585791fb9a695056e92adc0
Hash choices=sha256
CMSDigest=a9b0dd7d25ba9fb90f955011ffb78e5133c545e1d585791fb9a695056e92adc0
CMSDigestType=2
CDHash=a9b0dd7d25ba9fb90f955011ffb78e5133c545e1
Signature size=8978
Authority=Developer ID Application: Microsoft Corporation (UBF8T346G9)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Jan 25, 2021 at 5:54:12 PM
Info.plist=not bound
TeamIdentifier=UBF8T346G9
Runtime Version=10.15.0
Sealed Resources=none
Internal requirements count=1 size=172

cc @rbhanda @mmitche

sdmaclea commented 3 years ago

Hmm I downloaded using the link @richlander provided. https://download.visualstudio.microsoft.com/download/pr/3de2d949-fcb5-4586-a217-2c33854d295f/943f0d92252338e11fd11b002a3a3861/dotnet-sdk-5.0.103-osx-x64.tar.gz

mmitche commented 3 years ago

Where are the download.visualstudio.com links populated from?

rbhanda commented 3 years ago

https://github.com/dotnet/core/blob/master/release-notes/5.0/releases.json is the one which drives the downloads. These are generated during the release process and the bits are tested and picked from the release file drops.

leecow commented 3 years ago

I believe we only sign and notarize pkgs. @mmitche, can you confirm that?

mmitche commented 3 years ago

I believe that is correct. I don't think you can notarize a non-pkg.

sdmaclea commented 3 years ago

We need to create an issue for this somewhere.

It makes no sense for us to distribute unsigned macos tar.gz on our release download pages. They will trigger malicious code warnings on BigSur (at least on Apple Silicon)
We have the signed files in our installer pkg
Macos does have signing tools to sign files codesign -s <signing cert> <path-to-sign>

janvorli commented 3 years ago

I think it still makes sense to distribute macos tar.gz. We've been doing that forever and I was always using that package and never had any issues except of the M1. It still works on my Mac Mini x64 with BigSur installed and SIP enabled. Our builds use the .tar.gz packages too without problems.

sdmaclea commented 3 years ago

Ack. We should create an issue. So we can have a formal discussion in an appropriate place.

ViktorHofer commented 3 years ago

It makes no sense for us to distribute unsigned macos tar.gz on our release download pages. They will trigger malicious code warnings on BigSur (at least on Apple Silicon)

cc @mmitche

mmitche commented 3 years ago

I think this could be done in 5.0 reasonably easily, and I think it would apply to the current 6.0 process. Otherwise we could modify the post-build signing in 6.0 to handle tar.gz files.

deeprobin commented 2 years ago

Is this still relevant for the .NET 5 milestone? (The EOS for this version is already on May 8, 2022).

danmoseley commented 2 years ago

@richlander for that q.

jkotas commented 2 years ago

.NET 6 is the first fully supported SDK for Rosetta 2 emulation.

See https://github.com/dotnet/sdk/issues/22380 for details.

dotnet / runtime

Support .NET on Apple Silicon with Rosetta 2 emulation #44897