search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.97k stars 4.66k forks source link

Reloc failures with NativeAOT on Apple Silicon #67232

Closed am11 closed 2 years ago

am11 commented 2 years ago

I am trying to enable NativeAOT on OSX arm64. With this patch https://github.com/dotnet/runtime/compare/main...am11:feature/nativeaot/osx-arm64 (tested with both @GOTPAGE and @PAGE assembler directives), it builds the nupkg. Consuming that package results in the following errors during the ilc step:

# with `<add key="TestSource" value="/Users/am11/projects/runtime/artifacts/packages/Release/Shipping" />`
# in NuGet.config
$ dotnet nuget locals all --clear && rm -rf obj bin && dotnet publish --use-current-runtime -v:diag ...
... snip ...
21:06:05.007   1:7>Target "IlcCompile: (TargetId:181)" in file "/Users/am11/.nuget/packages/microsoft.dotnet.ilcompiler/7.0.0-dev/build/Microsoft.NETCore.Native.targets" from project "/Users/am11/projects/naot1/naot1.csproj" (target "LinkNative" depends on it):
                   Building target "IlcCompile" completely.
                   Output file "obj/release/net7.0/osx-arm64/native/naot1.o" does not exist.
                   Task "Message" skipped, due to false condition; ($(_BuildingInCompatibleMode) != 'true') was evaluated as (true != 'true').
                   Task "Message" (TaskId:126)
                     Task Parameter:Text=Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeCoreRT (TaskId:126)
                     Task Parameter:Importance=high (TaskId:126)
                     Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeCoreRT (TaskId:126)
                   Done executing task "Message". (TaskId:126)
                   Task "Exec" (TaskId:127)
                     Task Parameter:Command="/Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/tools/ilc" @"obj/release/net7.0/osx-arm64/native/naot1.ilc.rsp" (TaskId:127)
                     "/Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/tools/ilc" @"obj/release/net7.0/osx-arm64/native/naot1.ilc.rsp" (TaskId:127)
                     <unknown>:0: error: ADR/ADRP relocations must be GOT relative (TaskId:127)
                     <unknown>:0: error: unknown AArch64 fixup kind! (TaskId:127)
                     <unknown>:0: error: unknown AArch64 fixup kind! (TaskId:127)
                     <unknown>:0: error: fixup value out of range (TaskId:127)
                     <unknown>:0: error: ADR/ADRP relocations must be GOT relative (TaskId:127)
                     <unknown>:0: error: unknown AArch64 fixup kind! (TaskId:127)
                     <unknown>:0: error: unknown AArch64 fixup kind! (TaskId:127)
                     <unknown>:0: error: fixup value out of range (TaskId:127)
... repeats 1000s of times ...

somewhere after the objwriter has succeeded: https://github.com/dotnet/runtime/blob/071e772d9d3bd8b50a5380bce6214277a1e61c98/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ObjectWriter.cs#L1183 and before the clang command is executed. While the ilc task does not fail, MSBuild fails on the clang step:

                 Set Property: _IgnoreLinkerWarnings=false
                   Set Property: _IgnoreLinkerWarnings=true
                   Task "Exec" (TaskId:129)
                     Task Parameter:IgnoreStandardErrorWarningFormat=True (TaskId:129)
                     Task Parameter:Command=clang "obj/release/net7.0/osx-arm64/native/naot1.o" -o "bin/release/net7.0/osx-arm64/native/naot1" /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/sdk/libbootstrapper.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/sdk/libRuntime.WorkstationGC.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Globalization.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.IO.Compression.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Net.Security.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Security.Cryptography.Native.Apple.a -g -Wl,-rpath,'@executable_path' -lstdc++ -ldl -lm -lz -licucore -framework CoreFoundation -framework Foundation -framework Security -framework GSS (TaskId:129)
                     clang "obj/release/net7.0/osx-arm64/native/naot1.o" -o "bin/release/net7.0/osx-arm64/native/naot1" /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/sdk/libbootstrapper.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/sdk/libRuntime.WorkstationGC.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Globalization.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.IO.Compression.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Net.Security.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Security.Cryptography.Native.Apple.a -g -Wl,-rpath,'@executable_path' -lstdc++ -ldl -lm -lz -licucore -framework CoreFoundation -framework Foundation -framework Security -framework GSS (TaskId:129)
                     ld: malformed __LD,__compact_unwind section, bad length file 'obj/release/net7.0/osx-arm64/native/naot1.o' (TaskId:129)
                     clang: error: linker command failed with exit code 1 (use -v to see invocation) (TaskId:129)
21:06:12.873   1:7>/Users/am11/.nuget/packages/microsoft.dotnet.ilcompiler/7.0.0-dev/build/Microsoft.NETCore.Native.targets(337,5): error MSB3073: The command "clang "obj/release/net7.0/osx-arm64/native/naot1.o" -o "bin/release/net7.0/osx-arm64/native/naot1" /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/sdk/libbootstrapper.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/sdk/libRuntime.WorkstationGC.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Globalization.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.IO.Compression.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Net.Security.Native.a /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Security.Cryptography.Native.Apple.a -g -Wl,-rpath,'@executable_path' -lstdc++ -ldl -lm -lz -licucore -framework CoreFoundation -framework Foundation -framework Security -framework GSS" exited with code 1. [/Users/am11/projects/naot1/naot1.csproj]
                   Done executing task "Exec" -- FAILED. (TaskId:129)
21:06:12.873   1:7>Done building target "LinkNative" in project "naot1.csproj" -- FAILED.: (TargetId:182)

With objdump, that __LD,__compact_unwind section looks like:

Disassembly of section __LD,__compact_unwind:

00000000003b2858 <ltmp8>:
  3b2858: 40 4b 00 00   udf     #19264
  3b285c: 00 00 00 00   udf     #0
  3b2860: 74 00 00 00   udf     #116
  3b2864: 00 00 00 03   <unknown>
                ...
  3b2874: c0 4b 00 00   udf     #19392
  3b2878: 00 00 00 00   udf     #0
  3b287c: 74 00 00 00   udf     #116
  3b2880: 00 00 00 03   <unknown>
                ...
  3b2890: 40 4c 00 00   udf     #19520
  3b2894: 00 00 00 00   udf     #0
  3b2898: 74 00 00 00   udf     #116
  3b289c: 00 00 00 03   <unknown>
                ...
  3b28ac: c0 4c 00 00   udf     #19648
  3b28b0: 00 00 00 00   udf     #0
  3b28b4: 74 00 00 00   udf     #116
  3b28b8: 00 00 00 03   <unknown>
... repeats ...
MichalStrehovsky commented 2 years ago

Ah, so those messages are still generated by the object writer in ILC.

For example here: https://github.com/dotnet/llvm-project/blob/f1120a92d05f1c57e75af7d16504012570ef3409/llvm/lib/Target/AArch64/MCTargetDesc/AArch64MachObjectWriter.cpp#L102-L103.

Looks like we need to decide what kind of relocation to generate when we're generating it.

am11 commented 2 years ago

@MichalStrehovsky, thanks for the pointers. The managed part of objwriter API currently does not support VariantKind, so I updated objwriter: https://github.com/dotnet/llvm-project/pull/185. With the current state of that PR (+ https://github.com/dotnet/runtime/compare/main...am11:feature/nativeaot/osx-arm64), these two ilc errors are vanished:

error: fixup value out of range
error: ADR/ADRP relocations must be GOT relative

but this one remains:

error: unknown AArch64 fixup kind!

it is false return from https://github.com/dotnet/llvm-project/blob/f1120a92d05f1c57e75af7d16504012570ef3409/llvm/lib/Target/AArch64/MCTargetDesc/AArch64MachObjectWriter.cpp#L59. I noticed that in objwriter, we do not explicitly create a MCFixup for mach, looks like those are classified by AArch64AsmBackend. 💭

MichalStrehovsky commented 2 years ago

Really nice progress!

What is the reloc it's complaining about? Is it FK_PCRel_4 by any chance? We add special handling to it in our LLVM patch: https://github.com/dotnet/llvm-project/commit/67f5503adc0fe9d886a33df58f49a4b3e2b8c21e. Maybe having it take the same path as FK_Data_4 would work?

You'll know if the reloc got messed up if it ends up pointing at garbage after linking.

am11 commented 2 years ago

Those were indeed all FK_PCRel_4. 😄

Pushed a commit to treat it as FK_Data_4. With that ilc msbuild task completes without producing any diagnostics.

clang task, however, continues to fail (still complaining about __LD,__compact_unwind section, bad length file). Interestingly, now the __LD,__compact_unwind section has disappeared when I run with objdump -d (as it was showing previously), but it shows up with objdump --full-contents -d.

Running ld command with -v shows nothing useful:

 % "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" \
       -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib \
       -dynamic -arch arm64 -platform_version macos 12.0.0 12.1 \
       -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk \
       -o bin/release/net7.0/osx-arm64/native/naot1 \
       -L/usr/local/lib obj/release/net7.0/osx-arm64/native/naot1.o \
       /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/sdk/libbootstrapper.a \
       /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/sdk/libRuntime.WorkstationGC.a \
       /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Native.a \
       /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Globalization.Native.a \
       /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.IO.Compression.Native.a \
       /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Net.Security.Native.a \
       /Users/am11/.nuget/packages/runtime.osx-arm64.microsoft.dotnet.ilcompiler/7.0.0-dev/framework/libSystem.Security.Cryptography.Native.Apple.a \
       -rpath @executable_path -lc++ -ldl -lm -lz -licucore \
       -framework CoreFoundation -framework Foundation -framework Security -framework GSS \
       -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a \
       -v

@(#)PROGRAM:ld  PROJECT:ld64-711
BUILD 21:57:24 Nov 17 2021
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
Library search paths:
    /usr/local/lib
    /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib
Framework search paths:
    /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/
ld: malformed __LD,__compact_unwind section, bad length file 'obj/release/net7.0/osx-arm64/native/naot1.o'
MichalStrehovsky commented 2 years ago

Digging in Apple's source code, the error seems to be:

https://github.com/apple-oss-distributions/ld64/blob/dbf8f7feb5579761f1623b004bd468bdea7c6225/src/ld/parsers/macho_relocatable_file.cpp#L5631-L5632

the size of __compact_unwind section is not divisible by the size of the unwind entry. That's odd because we don't generate the Apple weird thing, we generate DWARF CFI.

However, looking at LLVM source code, I think this is kicking in:

https://github.com/dotnet/llvm-project/blob/f1120a92d05f1c57e75af7d16504012570ef3409/llvm/lib/MC/MCObjectFileInfo.cpp#L30-L32

And LLVM does generate something on our behalf. Probably broken from the sound of it.

I would dig around that - can we still make do without a __compact_unwind section? If it's not present in the executable, maybe ld would still do the right thing and convert it from CFI to the compact unwind scheme for us.

Maybe the right thing would be to start generating compact unwinding because Apple tends to unceremoniously cut off things they don't like anymore after a couple years of supporting both the thing they stopped liking and the new shiny thing. Unwinding codes are currently generated in RyuJIT.

am11 commented 2 years ago

Thanks, and nice detective work finding those links! (I was only searching across org:dotnet 😅)

Disabling compact unwind (https://github.com/dotnet/llvm-project/pull/185/commits/090a4651fd8f1ccc27bf5c87f8ec00a9861daf09) revealed some missing symbols errors. Those error messages were generous enough to point me to the missing C_FUNC(): https://github.com/am11/runtime/commit/3bcd7b6c (I previously fixed couple of those in https://github.com/am11/runtime/commit/b8cc922c79653098a1ad9d3eb64bec7c1d8db58e which were failing the runtime build, but the new ones only show up when consuming Microsoft.DotNet.ILCompiler package 🤷)

Looks like we are getting there. Next error is:

  ld: in section __DATA,.corert_eh_table reloc 0: unknown relocation type 15 file 'obj/release/net7.0/osx-arm64/native/naot1.o'

Apparently, reloc 15 is assigned to PPC_RELOC_LOCAL_SECTDIFF (suggesting something is off the kilter.. 👀).

MichalStrehovsky commented 2 years ago

.corert_eh_table uses the 32-bit relative relocs that would be translated to FK_PCRel_4 I think - so something might be going wrong around that.

Can you try updating this:

https://github.com/dotnet/runtime/blob/59bb459449107850a3a477aa8bba65c57bec9736/src/coreclr/tools/Common/TypeSystem/Common/TargetDetails.cs#L94-L100

and make it so that it returns false for ARM64 macOS? (use Architecture and OperatingSystem that is conveniently available on the class).

This will make the compiler avoid generating 32bit relative relocations in favor of full pointers. It will make the executables a bit bigger. We would want to try find a way to do 32bit relative relocs eventually, but let's first find out if that's really the problem.

am11 commented 2 years ago

Having SupportsRelativePointers to return false had no effect, getting the ditto ld error. :(

MichalStrehovsky commented 2 years ago

Ah, there's an extra code path that is not active for Wasm or CppCodegen (for which SupportsRelativePointers was introduced) - replace IMAGE_REL_BASED_RELPTR32 with IMAGE_REL_BASED_DIR64 in src\coreclr\tools\aot\ILCompiler.Compiler\Compiler\DependencyAnalysis\ObjectWriter.cs. This is nothing but a hack (it's going to crash if we take a GC or exception at runtime), but it might help narrowing down the problem/make further progress.

am11 commented 2 years ago

Thanks, this workaround worked for .corert_eh_table section. Now I am getting it from another section:

  ld: in section __TEXT,__const reloc 0: unknown relocation type 15 file 'obj/release/net7.0/osx-arm64/native/naot1.o'

I'll try to find out what is causing type 15 failure (to avoid workarounds).

MichalStrehovsky commented 2 years ago

I'll try to find out what is causing type 15 failure (to avoid workarounds).

Yeah, that sounds like a better plan. This workaround is starting to get out of hand (I think now it's relocs generated by RyuJIT).

I was able to make quick progress on issues like this in the past by reducing the problem into a ZeroSharp no-runtime size (https://github.com/MichalStrehovsky/zerosharp) repro case. The object files generated out of that are just a couple kilobytes in size and it's easier to trace through the problematic code within the compiler with that. But it's not mandatory to go in that direction, just a possible avenue if too many things are happening in a full Hello World.

am11 commented 2 years ago

Pushed a commit to objwriter (https://github.com/dotnet/llvm-project/pull/185/commits/7280b550bb8ac3cce72a1ee288dc744b1ab9b1b6) which fixes type 15 error. After that dotnet publish produced the binary successfully but it does not print Hello World! yet. :)

 % lldb bin/release/net7.0/osx-arm64/publish/naot1                
Added Microsoft public symbol server
Added symbol directory path: /usr/local/share/dotnet/shared/Microsoft.NETCore.App/6.0.2
Added symbol directory path: /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Host.osx-arm64/6.0.2/runtimes/osx-arm64/native
(lldb) target create "../naot1/bin/release/net7.0/osx-arm64/publish/naot1"
Current executable set to '/Users/am11/projects/naot1/bin/release/net7.0/osx-arm64/publish/naot1' (arm64).
(lldb) r
Process 38291 launched: '/Users/am11/projects/naot1/bin/release/net7.0/osx-arm64/publish/naot1' (arm64)
Process 38291 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x100460988)
    frame #0: 0x0000000100460988 naot1`tls_CurrentThread
naot1`tls_CurrentThread:
->  0x100460988 <+0>:  ldp    x16, x17, [x9, #-0xd0]
    0x10046098c <+4>:  udf    #0x1
    0x100460990 <+8>:  udf    #0x102
    0x100460994 <+12>: udf    #0x0
Target 0: (naot1) stopped.
(lldb) bt all
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x100460988)
  * frame #0: 0x0000000100460988 naot1`tls_CurrentThread
    frame #1: 0x00000001001a6db0 naot1`InitializeModules + 80
    frame #2: 0x0000000100006438 naot1`main [inlined] InitializeRuntime() at main.cpp:169:5 [opt]
    frame #3: 0x00000001000063ac naot1`main(argc=1, argv=0x000000016fdff798) at main.cpp:201:19 [opt]
    frame #4: 0x0000000100c350f4 dyld`start + 520
  thread #2
    frame #0: 0x00000001a96f1eac libsystem_kernel.dylib`mach_absolute_time + 108
    frame #1: 0x00000001a96f3838 libsystem_kernel.dylib`__commpage_gettimeofday_internal + 44
    frame #2: 0x00000001a95f9534 libsystem_c.dylib`gettimeofday + 52
    frame #3: 0x000000010004fcb4 naot1`::QueryPerformanceCounter(lpPerformanceCount=0x000000016fe86f68) at PalRedhawkUnix.cpp:1090:9 [opt]
    frame #4: 0x0000000100012090 naot1`EnsureYieldProcessorNormalizedInitialized() [inlined] PalQueryPerformanceCounter(arg1=0x000000016fe86f68) at PalRedhawkFunctions.h:131:12 [opt]
    frame #5: 0x0000000100012088 naot1`EnsureYieldProcessorNormalizedInitialized() at yieldprocessornormalized.cpp:76:9 [opt]
    frame #6: 0x0000000100012024 naot1`EnsureYieldProcessorNormalizedInitialized() at yieldprocessornormalized.cpp:118:9 [opt]
    frame #7: 0x000000010000801c naot1`FinalizerStart(pContext=0x0000600003000090) at FinalizerHelpers.cpp:54:5 [opt]
    frame #8: 0x00000001a972d240 libsystem_pthread.dylib`_pthread_start + 148
MichalStrehovsky commented 2 years ago

Great progress!

If I'm looking at the right thing, tls_currentThread is data, not code, so I guess we shouldn't be running it. But seeing a problem around it is not completely surprising since TLS access is likely going to be different on macOS than it is on Linux. The fix will probably be around the INLINE_GET_TLS_VAR macro in src\coreclr\nativeaot\Runtime. It should expand to however threadlocal statics are accessed on ARM64. There already is an if APPLE on the x64 version of the macro.

am11 commented 2 years ago

Yup, that was it: https://github.com/am11/runtime/commit/5106fc986382dae29aeaf93402f0f3dea408bc7d (based on ELF sequence vs. MachO)

It has moved the needle a bit; next up is RhpNewArray:

(lldb) r
Process 89591 launched: '/Users/am11/projects/naot1/bin/release/net7.0/osx-arm64/publish/naot1' (arm64)
This version of LLDB has no plugin for the mipsassem language. Inspection of frame variables will be limited.
Process 89591 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x1a9734530)
    frame #0: 0x000000010005c750 naot1`RhpNewArray at AllocFast.S:213
   210          ldr         x12, [x3, #OFFSETOF__Thread__m_alloc_context__alloc_ptr]
   211  
   212          // Update the alloc pointer to account for the allocation.
-> 213          str         x2, [x3, #OFFSETOF__Thread__m_alloc_context__alloc_ptr]
   214  
   215          // Set the new objects MethodTable pointer and element count.
   216          str         x0, [x12, #OFFSETOF__Object__m_pEEType]
Target 0: (naot1) stopped.

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x1a9734530)
  * frame #0: 0x000000010005c750 naot1`RhpNewArray at AllocFast.S:213
    frame #1: 0x00000001001a7114 naot1`S_P_CoreLib_Internal_Runtime_CompilerHelpers_StartupCodeHelpers__CreateTypeManagers + 100
    frame #2: 0x00000001001a6db0 naot1`InitializeModules + 80
    frame #3: 0x0000000100006458 naot1`main [inlined] InitializeRuntime() at main.cpp:169:5 [opt]
    frame #4: 0x00000001000063cc naot1`main(argc=1, argv=0x000000016fdff7a0) at main.cpp:201:19 [opt]
    frame #5: 0x0000000100c350f4 dyld`start + 520

(lldb) register read
General Purpose Registers:
        x0 = 0x0000000100f04290
        x1 = 0x0000000000000001
        x2 = 0xd53bd071f9400430
        x3 = 0x00000001a9734530  libdyld.dylib`tlv_get_addr
        x4 = 0x000000000000000a
        x5 = 0x0000000000000000
        x6 = 0x0000000000000002
        x7 = 0x0000000000000000
        x8 = 0x00000000ffffffff
        x9 = 0x0000000000000000
       x10 = 0x0000000000000070
       x11 = 0x0000000000000001
       x12 = 0xd53bd071f9400410
       x13 = 0x0000000001dfb800
       x14 = 0x0000000001c00000
       x15 = 0x0000000000000044
       x16 = 0x0000000000000000
       x17 = 0x0000000100f04290
       x18 = 0x0000000000000000
       x19 = 0x0000000000000001
       x20 = 0x0000000100460968  naot1`__Module
       x21 = 0x0000000100000000  naot1`_mh_execute_header
       x22 = 0x00000001003773a0  naot1`c_classlibFunctions
       x23 = 0x000000000000000a
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016fdff570
        lr = 0x000000010005c738  naot1`RhpNewArray + 56
        sp = 0x000000016fdff560
        pc = 0x000000010005c750  naot1`RhpNewArray + 80
      cpsr = 0x80001000

# a weird "error: warning: warning:" from printer
(lldb) p OFFSETOF__Thread__m_alloc_context__alloc_ptr
error: warning: warning: got name from symbols: OFFSETOF__Thread__m_alloc_context__alloc_ptr
error: <user expression 8>:1:1: reference to 'OFFSETOF__Thread__m_alloc_context__alloc_ptr' is ambiguous
OFFSETOF__Thread__m_alloc_context__alloc_ptr
^
note: candidate found by name lookup is 'OFFSETOF__Thread__m_alloc_context__alloc_ptr'

note: candidate found by name lookup is 'OFFSETOF__Thread__m_alloc_context__alloc_ptr'

# but this suggests offset is zero
(lldb) image lookup -n OFFSETOF__Thread__m_alloc_context__alloc_ptr
1 match found in /Users/am11/projects/naot1/bin/release/net7.0/osx-arm64/publish/naot1:
        Address: 0x0000000000000000 (0x0000000000000000)
        Summary: 0x0000000000000000
MichalStrehovsky commented 2 years ago

That looks related to the TLS access. We're here:

https://github.com/dotnet/runtime/blob/9b2e2a830a4e2e67c920aa200329533baba5c363/src/coreclr/nativeaot/Runtime/arm64/AllocFast.S#L193-L213

My suspicion is that INLINE_GETTHREAD loaded a bogus address into x3. It's supposed to load the tls_CurrentThread thread-local static.

https://github.com/dotnet/runtime/blob/9b2e2a830a4e2e67c920aa200329533baba5c363/src/coreclr/nativeaot/Runtime/unix/unixasmmacrosarm64.inc#L215-L217

I would put a breakpoint here:

https://github.com/dotnet/runtime/blob/9b2e2a830a4e2e67c920aa200329533baba5c363/src/coreclr/nativeaot/Runtime/threadstore.inl#L6-L10

and see what value the variable has (and how the compiler got to it in assembly). Then compare with what INLINE_GETTHREAD came up with. (Make sure you're looking at the same thread, we already have the finalizer thread running at this point in startup).

am11 commented 2 years ago

Ah, right, had to move x0 to the target register and also pop stack before endm for APPLE branch: https://github.com/am11/runtime/commit/750fc8349ce41a66998cd5a847d2e22c979c285e. Thanks for the debugging pointers, I just printed its address with p (Thread *) &tls_CurrentThread and then compared what's the effect on x3 before and after Alloc.S:196.

Now we are in the managed main and getting to the code emitted via objwriter:

(lldb) target create "../naot1/bin/release/net7.0/osx-arm64/native/naot1"
Current executable set to '/Users/am11/projects/naot1/bin/release/net7.0/osx-arm64/native/naot1' (arm64).

(lldb) r
Process 86586 launched: '/Users/am11/projects/naot1/bin/release/net7.0/osx-arm64/native/naot1' (arm64)
Process 86586 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x00000001001fa8a4 naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 52
naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create:
->  0x1001fa8a4 <+52>: ldr    x1, [x0]
    0x1001fa8a8 <+56>: ldr    x1, [x1, #0x1e8]
    0x1001fa8ac <+60>: blr    x1
    0x1001fa8b0 <+64>: bl     0x1001a3b10               ; S_P_CoreLib_Internal_IntrinsicSupport_EqualityComparerHelpers__GetComparer
Target 0: (naot1) stopped.

(lldb) bt all
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x00000001001fa8a4 naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 52
    frame #1: 0x0000000100190dd4 naot1`S_P_CoreLib_System_Collections_Generic_NonRandomizedStringEqualityComparer___cctor + 36
    frame #2: 0x000000010019722c naot1`S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun + 204
    frame #3: 0x00000001001970e4 naot1`S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnGCStaticBase + 20
    frame #4: 0x0000000100190d48 naot1`S_P_CoreLib_System_Collections_Generic_NonRandomizedStringEqualityComparer__GetStringComparer + 24
    frame #5: 0x000000010020cc80 naot1`S_P_CoreLib_System_Collections_Generic_Dictionary_2<System___Canon__System___Canon>___ctor_2 + 128
    frame #6: 0x00000001000e2a04 naot1`S_P_CoreLib_System_AppContext__SetData + 84
    frame #7: 0x00000001002125a0 naot1`Internal_CompilerGenerated__Module___SetAppContextSwitches + 32
    frame #8: 0x0000000100212734 naot1`__managed__Main + 228
    frame #9: 0x0000000100006454 naot1`main(argc=1, argv=0x000000016fdff798) at main.cpp:205:18 [opt]
    frame #10: 0x0000000100c350f4 dyld`start + 520
  thread #2
    frame #0: 0x00000001a96f1eac libsystem_kernel.dylib`mach_absolute_time + 108
    frame #1: 0x00000001a96f3838 libsystem_kernel.dylib`__commpage_gettimeofday_internal + 44
    frame #2: 0x00000001a95f9534 libsystem_c.dylib`gettimeofday + 52
    frame #3: 0x000000010004fcc4 naot1`::QueryPerformanceCounter(lpPerformanceCount=0x000000016fe86f68) at PalRedhawkUnix.cpp:1090:9 [opt]
    frame #4: 0x00000001000120a0 naot1`EnsureYieldProcessorNormalizedInitialized() [inlined] PalQueryPerformanceCounter(arg1=0x000000016fe86f68) at PalRedhawkFunctions.h:131:12 [opt]
    frame #5: 0x0000000100012098 naot1`EnsureYieldProcessorNormalizedInitialized() at yieldprocessornormalized.cpp:76:9 [opt]
    frame #6: 0x0000000100012034 naot1`EnsureYieldProcessorNormalizedInitialized() at yieldprocessornormalized.cpp:118:9 [opt]
    frame #7: 0x000000010000802c naot1`FinalizerStart(pContext=0x0000600003000090) at FinalizerHelpers.cpp:54:5 [opt]
    frame #8: 0x00000001a972d240 libsystem_pthread.dylib`_pthread_start + 148

(lldb) register read
General Purpose Registers:
        x0 = 0x0000000000000000
        x1 = 0x0000000100373628  (void *)0x0000000100469238: __writableDataString
        x2 = 0x00000001003eeaf0  __TypeThreadStaticIndexS_P_CoreLib_System_Threading_ManagedThreadId
        x3 = 0x0000000000000018
        x4 = 0x0000000000000000
        x5 = 0x0000000101800000
        x6 = 0x0000000000000007
        x7 = 0x0000000000000000
        x8 = 0x0000000100ec11f0
        x9 = 0x0000600000004090
       x10 = 0x00000001a9734530  libdyld.dylib`tlv_get_addr
       x11 = 0x0000000000000001
       x12 = 0x000000010045f1e8  g_ephemeral_high
       x13 = 0x0000000101807fd0
       x14 = 0x0000000101806608
       x15 = 0x0000000101806620
       x16 = 0x0000000000000000
       x17 = 0x0000000100f04290
       x18 = 0x0000000000000000
       x19 = 0x000000010041bf70  vtable for S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<String>
       x20 = 0x00000001198061d8
       x21 = 0x00000001198061e0
       x22 = 0x0000000101806440
       x23 = 0x0000000119804820
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016fdff420
        lr = 0x00000001001fa8a4  naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 52
        sp = 0x000000016fdff420
        pc = 0x00000001001fa8a4  naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 52
      cpsr = 0x60001000
MichalStrehovsky commented 2 years ago

This doesn't ring any bell - can you paste the full disassembly of the faulting method? It's not clear where we are. I'm guessing that this is an attempt to do a virtual call to access the TypeHandle property here:

https://github.com/dotnet/runtime/blob/07e87bc4cb0358f57e7116e047f7d2017b049cf9/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Collections/Generic/EqualityComparer.CoreRT.cs#L30

...but the System.Type we got out of the typeof is bogus. Right before that you should see a call to this method:

https://github.com/dotnet/runtime/blob/07e87bc4cb0358f57e7116e047f7d2017b049cf9/src/coreclr/nativeaot/System.Private.CoreLib/src/Internal/Runtime/CompilerHelpers/LdTokenHelpers.cs#L34-L37

Validate that the parameter to the method is correct (it should be a MethodTable* and should have an associated textual symbol) and then try to step through what it's doing.

Might be easier to debug through this with optimizations off (drop the -O parameter from the ilc command line or dotnet publish as Debug).

am11 commented 2 years ago

With debug, here is the full disassembly at IP/PC (0x1002f0bb8 in the middle of disassembly):

(lldb) r
Process 88778 launched: '/Users/am11/projects/naot1/bin/Debug/net7.0/osx-arm64/publish/naot1' (arm64)
Process 88778 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x00000001002f0bb8 naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 168
naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create:
->  0x1002f0bb8 <+168>: ldr    wzr, [x0]
    0x1002f0bbc <+172>: blr    x1
    0x1002f0bc0 <+176>: str    x0, [x29, #0x48]
    0x1002f0bc4 <+180>: ldr    x0, [x29, #0x48]
Target 0: (naot1) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x00000001002f0bb8 naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 168
    frame #1: 0x00000001002f0c74 naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__get_Default + 68
    frame #2: 0x000000010023dec4 naot1`S_P_CoreLib_System_Collections_Generic_NonRandomizedStringEqualityComparer___cctor + 36
    frame #3: 0x0000000100247ba0 naot1`S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun + 384
    frame #4: 0x0000000100247988 naot1`S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnGCStaticBase + 24
    frame #5: 0x000000010023de28 naot1`S_P_CoreLib_System_Collections_Generic_NonRandomizedStringEqualityComparer__GetStringComparer + 40
    frame #6: 0x000000010030e32c naot1`S_P_CoreLib_System_Collections_Generic_Dictionary_2<System___Canon__System___Canon>___ctor_2 + 268
    frame #7: 0x000000010030e1dc naot1`S_P_CoreLib_System_Collections_Generic_Dictionary_2<System___Canon__System___Canon>___ctor + 28
    frame #8: 0x0000000100123414 naot1`S_P_CoreLib_System_AppContext__SetData + 116
    frame #9: 0x0000000100317a9c naot1`Internal_CompilerGenerated__Module___SetAppContextSwitches + 28
    frame #10: 0x0000000100317bbc naot1`__managed__Main + 60
    frame #11: 0x000000010000df64 naot1`main(argc=1, argv=0x000000016fdff7a0) at main.cpp:205:18 [opt]
    frame #12: 0x0000000100f190f4 dyld`start + 520
(lldb) disassemble -a 0x1002f0bb8
naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create:
    0x1002f0b10 <+0>:   stp    x29, x30, [sp, #-0x90]!
    0x1002f0b14 <+4>:   mov    x29, sp
    0x1002f0b18 <+8>:   add    x9, x29, #0x28            ; =0x28 
    0x1002f0b1c <+12>:  movi.16b v16, #0x0
    0x1002f0b20 <+16>:  stp    q16, q16, [x9]
    0x1002f0b24 <+20>:  stp    q16, q16, [x9, #0x20]
    0x1002f0b28 <+24>:  stp    xzr, xzr, [x9, #0x40]
    0x1002f0b2c <+28>:  str    xzr, [x9, #0x50]
    0x1002f0b30 <+32>:  str    x0, [x29, #0x88]
    0x1002f0b34 <+36>:  str    x0, [x29, #0x80]
    0x1002f0b38 <+40>:  ldr    x0, [x29, #0x80]
    0x1002f0b3c <+44>:  bl     0x10000b380               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_GCStaticBase_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>
    0x1002f0b40 <+48>:  add    x0, x0, #0x8              ; =0x8 
    0x1002f0b44 <+52>:  str    x0, [x29, #0x78]
    0x1002f0b48 <+56>:  ldr    x0, [x29, #0x80]
    0x1002f0b4c <+60>:  bl     0x10000bd54               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_TypeHandle_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>
    0x1002f0b50 <+64>:  bl     0x1002f0af0               ; S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__get_SupportsGenericIEquatableInterfaces
    0x1002f0b54 <+68>:  str    w0, [x29, #0x74]
    0x1002f0b58 <+72>:  ldr    x0, [x29, #0x78]
    0x1002f0b5c <+76>:  str    x0, [x29, #0x68]
    0x1002f0b60 <+80>:  ldr    w0, [x29, #0x74]
    0x1002f0b64 <+84>:  cbnz   w0, 0x1002f0b88           ; <+120>
    0x1002f0b68 <+88>:  nop    
    0x1002f0b6c <+92>:  nop    
    0x1002f0b70 <+96>:  nop    
    0x1002f0b74 <+100>: nop    
    0x1002f0b78 <+104>: nop    
    0x1002f0b7c <+108>: ldr    x0, [x29, #0x68]
    0x1002f0b80 <+112>: str    x0, [x29, #0x28]
    0x1002f0b84 <+116>: b      0x1002f0b84               ; <+116>
    0x1002f0b88 <+120>: ldr    x0, [x29, #0x80]
    0x1002f0b8c <+124>: bl     0x10000bd60               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_TypeHandle_T_System___Canon
    0x1002f0b90 <+128>: bl     0x100261960               ; S_P_CoreLib_Internal_Runtime_CompilerHelpers_LdTokenHelpers__GetRuntimeTypeHandle
    0x1002f0b94 <+132>: str    x0, [x29, #0x60]
    0x1002f0b98 <+136>: ldr    x0, [x29, #0x60]
    0x1002f0b9c <+140>: bl     0x100140890               ; S_P_CoreLib_System_Type__GetTypeFromHandle
    0x1002f0ba0 <+144>: str    x0, [x29, #0x58]
    0x1002f0ba4 <+148>: adrp   x0, -747
    0x1002f0ba8 <+152>: add    x0, x0, #0x4a8            ; =0x4a8 
    0x1002f0bac <+156>: str    x0, [x29, #0x50]
    0x1002f0bb0 <+160>: ldr    x0, [x29, #0x58]
    0x1002f0bb4 <+164>: ldr    x1, [x29, #0x50]
->  0x1002f0bb8 <+168>: ldr    wzr, [x0]
    0x1002f0bbc <+172>: blr    x1
    0x1002f0bc0 <+176>: str    x0, [x29, #0x48]
    0x1002f0bc4 <+180>: ldr    x0, [x29, #0x48]
    0x1002f0bc8 <+184>: bl     0x1002596d0               ; S_P_CoreLib_Internal_IntrinsicSupport_EqualityComparerHelpers__GetComparer
    0x1002f0bcc <+188>: str    x0, [x29, #0x40]
    0x1002f0bd0 <+192>: ldr    x0, [x29, #0x80]
    0x1002f0bd4 <+196>: bl     0x10000c954               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_MethodDictionary_S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>>
    0x1002f0bd8 <+200>: str    x0, [x29, #0x20]
    0x1002f0bdc <+204>: ldr    x0, [x29, #0x20]
    0x1002f0be0 <+208>: ldr    x1, [x29, #0x40]
    0x1002f0be4 <+212>: bl     0x100370490               ; S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<System___Canon>
    0x1002f0be8 <+216>: str    x0, [x29, #0x38]
    0x1002f0bec <+220>: ldr    x0, [x29, #0x80]
    0x1002f0bf0 <+224>: bl     0x10000c948               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_MethodDictionary_S_P_CoreLib_System_Threading_Interlocked__CompareExchange_3<S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>>
    0x1002f0bf4 <+228>: str    x0, [x29, #0x18]
    0x1002f0bf8 <+232>: ldr    x0, [x29, #0x18]
    0x1002f0bfc <+236>: ldr    x1, [x29, #0x68]
    0x1002f0c00 <+240>: ldr    x2, [x29, #0x38]
    0x1002f0c04 <+244>: mov    x3, xzr
    0x1002f0c08 <+248>: bl     0x10036f770               ; S_P_CoreLib_System_Threading_Interlocked__CompareExchange_3<System___Canon>
    0x1002f0c0c <+252>: str    x0, [x29, #0x30]
    0x1002f0c10 <+256>: nop    
    0x1002f0c14 <+260>: ldr    x0, [x29, #0x80]
    0x1002f0c18 <+264>: bl     0x10000b380               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_GCStaticBase_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>
    0x1002f0c1c <+268>: ldr    x0, [x0, #0x8]
    0x1002f0c20 <+272>: ldp    x29, x30, [sp], #0x90
    0x1002f0c24 <+276>: ret    
    0x1002f0c28 <+280>: udf    #0x0
    0x1002f0c2c <+284>: udf    #0x0

(lldb) register read
General Purpose Registers:
        x0 = 0x0000000000000000
        x1 = 0x00000001000054a8  naot1`__VirtualCall_S_P_CoreLib_System_Type__get_TypeHandle
        x2 = 0x000000000000000a
        x3 = 0x0000000102006668
        x4 = 0x0000000000000020
        x5 = 0x0000000000000003
        x6 = 0x0000000000000007
        x7 = 0x0000000000000000
        x8 = 0x00000001011a11f0
        x9 = 0x000000016fdff218
       x10 = 0x00000001a9734530  libdyld.dylib`tlv_get_addr
       x11 = 0x0000000000000001
       x12 = 0x00000001005fe7a8  g_ephemeral_high
       x13 = 0x0000000102007fd0
       x14 = 0x0000000102006638
       x15 = 0x0000000102006650
       x16 = 0x0000000000000000
       x17 = 0x0000000101204290
       x18 = 0x0000000000000000
       x19 = 0x000000016fdff7a0
       x20 = 0x0000000000000001
       x21 = 0x0000000100000000  naot1`_mh_execute_header
       x22 = 0x0000000000000000
       x23 = 0x0000000000000000
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016fdff290
        lr = 0x00000001002f0ba0  naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 144
        sp = 0x000000016fdff290
        pc = 0x00000001002f0bb8  naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 168
      cpsr = 0x20001000

x0 is null.

BTW, runtime's build.sh also produced ~/projects/runtime/artifacts/packages/Release/Shipping/Microsoft.DotNet.ILCompiler.7.0.0-dev.symbols.nupkg, but I am not sure if dotnet-sos, dotnet-symbol and friends recognize it OOTB (there were few exports which were required for singlefilehost to work with SOS: https://github.com/dotnet/runtime/commit/b0621e79cc2fe988875dd97a46f131439bdce636 and https://github.com/dotnet/runtime/commit/8c6e3e995af2714bc4c35086afce0cdbf449df1d. That would make things super easy to debug.

MichalStrehovsky commented 2 years ago

I'm still looking at the disassembly, but while I'm doing that - the compiler produces DWARF debug information so you should be getting line numbers and local variables. It's supposed to debug like C++. Make sure ILC is invoked with the -g option (it should be unless opted out).

am11 commented 2 years ago

Yup, ./obj/Debug/net7.0/osx-arm64/native/naot1.ilc.rsp has -g. So the output in lldb is complete/expected. 👍

MichalStrehovsky commented 2 years ago

Can you dump what's in x0 at the spot where we call S_P_CoreLib_System_Type__GetTypeFromHandle? It should be an address that has a symbol associated with it (image lookup -va the_address_in_x0 should show a symbol for it). If there's no symbol, it's suspicious - it should be pointing to a MethodTable so you can cast it to MethodTable* and dump the contents to see if it looks legit. If it has a symbol, it's legit for sure.

If x0 is already bogus at that point, check x0 before the call to __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_TypeHandle_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>. That one should also be a MethodTable with a symbol associated.

Yup, ./obj/Debug/net7.0/osx-arm64/native/naot1.ilc.rsp has -g. So the output in lldb is complete/expected. 👍

Just to double check - do you get line debugging and local variables as well?

am11 commented 2 years ago

vtable for String is the name:

(lldb) r
Process 89722 launched: '/Users/am11/projects/naot1/bin/Debug/net7.0/osx-arm64/publish/naot1' (arm64)
Process 89722 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x00000001002f0bb8 naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 168
naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create:
->  0x1002f0bb8 <+168>: ldr    wzr, [x0]
    0x1002f0bbc <+172>: blr    x1
    0x1002f0bc0 <+176>: str    x0, [x29, #0x48]
    0x1002f0bc4 <+180>: ldr    x0, [x29, #0x48]
Target 0: (naot1) stopped.

(lldb) disassemble -a 0x1002f0bb8
naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create:
    0x1002f0b10 <+0>:   stp    x29, x30, [sp, #-0x90]!
    0x1002f0b14 <+4>:   mov    x29, sp
    0x1002f0b18 <+8>:   add    x9, x29, #0x28            ; =0x28 
    0x1002f0b1c <+12>:  movi.16b v16, #0x0
    0x1002f0b20 <+16>:  stp    q16, q16, [x9]
    0x1002f0b24 <+20>:  stp    q16, q16, [x9, #0x20]
    0x1002f0b28 <+24>:  stp    xzr, xzr, [x9, #0x40]
    0x1002f0b2c <+28>:  str    xzr, [x9, #0x50]
    0x1002f0b30 <+32>:  str    x0, [x29, #0x88]
    0x1002f0b34 <+36>:  str    x0, [x29, #0x80]
    0x1002f0b38 <+40>:  ldr    x0, [x29, #0x80]
    0x1002f0b3c <+44>:  bl     0x10000b380               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_GCStaticBase_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>
    0x1002f0b40 <+48>:  add    x0, x0, #0x8              ; =0x8 
    0x1002f0b44 <+52>:  str    x0, [x29, #0x78]
    0x1002f0b48 <+56>:  ldr    x0, [x29, #0x80]
    0x1002f0b4c <+60>:  bl     0x10000bd54               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_TypeHandle_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>
    0x1002f0b50 <+64>:  bl     0x1002f0af0               ; S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__get_SupportsGenericIEquatableInterfaces
    0x1002f0b54 <+68>:  str    w0, [x29, #0x74]
    0x1002f0b58 <+72>:  ldr    x0, [x29, #0x78]
    0x1002f0b5c <+76>:  str    x0, [x29, #0x68]
    0x1002f0b60 <+80>:  ldr    w0, [x29, #0x74]
    0x1002f0b64 <+84>:  cbnz   w0, 0x1002f0b88           ; <+120>
    0x1002f0b68 <+88>:  nop    
    0x1002f0b6c <+92>:  nop    
    0x1002f0b70 <+96>:  nop    
    0x1002f0b74 <+100>: nop    
    0x1002f0b78 <+104>: nop    
    0x1002f0b7c <+108>: ldr    x0, [x29, #0x68]
    0x1002f0b80 <+112>: str    x0, [x29, #0x28]
    0x1002f0b84 <+116>: b      0x1002f0b84               ; <+116>
    0x1002f0b88 <+120>: ldr    x0, [x29, #0x80]
    0x1002f0b8c <+124>: bl     0x10000bd60               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_TypeHandle_T_System___Canon
    0x1002f0b90 <+128>: bl     0x100261960               ; S_P_CoreLib_Internal_Runtime_CompilerHelpers_LdTokenHelpers__GetRuntimeTypeHandle
    0x1002f0b94 <+132>: str    x0, [x29, #0x60]
    0x1002f0b98 <+136>: ldr    x0, [x29, #0x60]
    0x1002f0b9c <+140>: bl     0x100140890               ; S_P_CoreLib_System_Type__GetTypeFromHandle
    0x1002f0ba0 <+144>: str    x0, [x29, #0x58]
    0x1002f0ba4 <+148>: adrp   x0, -747
    0x1002f0ba8 <+152>: add    x0, x0, #0x4a8            ; =0x4a8 
    0x1002f0bac <+156>: str    x0, [x29, #0x50]
    0x1002f0bb0 <+160>: ldr    x0, [x29, #0x58]
    0x1002f0bb4 <+164>: ldr    x1, [x29, #0x50]
->  0x1002f0bb8 <+168>: ldr    wzr, [x0]
    0x1002f0bbc <+172>: blr    x1
    0x1002f0bc0 <+176>: str    x0, [x29, #0x48]
    0x1002f0bc4 <+180>: ldr    x0, [x29, #0x48]
    0x1002f0bc8 <+184>: bl     0x1002596d0               ; S_P_CoreLib_Internal_IntrinsicSupport_EqualityComparerHelpers__GetComparer
    0x1002f0bcc <+188>: str    x0, [x29, #0x40]
    0x1002f0bd0 <+192>: ldr    x0, [x29, #0x80]
    0x1002f0bd4 <+196>: bl     0x10000c954               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_MethodDictionary_S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>>
    0x1002f0bd8 <+200>: str    x0, [x29, #0x20]
    0x1002f0bdc <+204>: ldr    x0, [x29, #0x20]
    0x1002f0be0 <+208>: ldr    x1, [x29, #0x40]
    0x1002f0be4 <+212>: bl     0x100370490               ; S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<System___Canon>
    0x1002f0be8 <+216>: str    x0, [x29, #0x38]
    0x1002f0bec <+220>: ldr    x0, [x29, #0x80]
    0x1002f0bf0 <+224>: bl     0x10000c948               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_MethodDictionary_S_P_CoreLib_System_Threading_Interlocked__CompareExchange_3<S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>>
    0x1002f0bf4 <+228>: str    x0, [x29, #0x18]
    0x1002f0bf8 <+232>: ldr    x0, [x29, #0x18]
    0x1002f0bfc <+236>: ldr    x1, [x29, #0x68]
    0x1002f0c00 <+240>: ldr    x2, [x29, #0x38]
    0x1002f0c04 <+244>: mov    x3, xzr
    0x1002f0c08 <+248>: bl     0x10036f770               ; S_P_CoreLib_System_Threading_Interlocked__CompareExchange_3<System___Canon>
    0x1002f0c0c <+252>: str    x0, [x29, #0x30]
    0x1002f0c10 <+256>: nop    
    0x1002f0c14 <+260>: ldr    x0, [x29, #0x80]
    0x1002f0c18 <+264>: bl     0x10000b380               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_GCStaticBase_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>
    0x1002f0c1c <+268>: ldr    x0, [x0, #0x8]
    0x1002f0c20 <+272>: ldp    x29, x30, [sp], #0x90
    0x1002f0c24 <+276>: ret    
    0x1002f0c28 <+280>: udf    #0x0
    0x1002f0c2c <+284>: udf    #0x0

(lldb) b -a 0x1002f0b9c
Breakpoint 1: where = naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 140, address = 0x00000001002f0b9c

(lldb) r
There is a running process, kill it and restart?: [Y/n] Y
Process 89722 exited with status = 9 (0x00000009) 
Process 89732 launched: '/Users/am11/projects/naot1/bin/Debug/net7.0/osx-arm64/publish/naot1' (arm64)
Process 89732 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x00000001002f0b9c naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 140
naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create:
->  0x1002f0b9c <+140>: bl     0x100140890               ; S_P_CoreLib_System_Type__GetTypeFromHandle
    0x1002f0ba0 <+144>: str    x0, [x29, #0x58]
    0x1002f0ba4 <+148>: adrp   x0, -747
    0x1002f0ba8 <+152>: add    x0, x0, #0x4a8            ; =0x4a8 
Target 0: (naot1) stopped.

(lldb) register read x0
      x0 = 0x00000001005a0620  vtable for String

(lldb) image lookup -va 0x00000001005a0620
      Address: naot1[0x00000001005a0620] (naot1.__DATA.__data + 297760)
      Summary: vtable for String
       Module: file = "/Users/am11/projects/naot1/bin/Debug/net7.0/osx-arm64/publish/naot1", arch = "arm64"
       Symbol: id = {0x00008410}, range = [0x00000001005a0620-0x00000001005a0698), name="vtable for String", mangled="_ZTV6String"
am11 commented 2 years ago

Just to double check - do you get line debugging and local variables as well?

Yup, previously I set breakpoint in Alloc.S (in the published app with release configuration): b Alloc.S:196 Alloc.S and friends seem to have embedded source with actual code comments.. which is probably too much for release configuration. 😅

MichalStrehovsky commented 2 years ago

Vtable for string is expected. Can step through the process of converting that to a System.Type instance? It should be pretty short. The MethodTable has a pointer to a writable chunk of memory (might be a problem with that reloc?). The writable chunk of memory has a GCHandle to the System.Type instance. It will likely be uninitialized at this point. We should return a valid object instance.

am11 commented 2 years ago

The intrinsic function GetTypeFromEETypePtr was inlined, stepping into visited this method: https://github.com/dotnet/runtime/blob/07e87bc4cb0358f57e7116e047f7d2017b049cf9/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Type.CoreRT.cs#L23 and it jumped back to the caller after failing that null check. It indeed looks related to the virtual call (or perhaps the interlock exchange in GetTypeFromEETypePtrSlow?):

    0x1002f0b9c <+140>: bl     0x100140890               ; S_P_CoreLib_System_Type__GetTypeFromHandle
      x0 = 0x00000001005a0620  vtable for String

    0x1002f0ba0 <+144>: str    x0, [x29, #0x58]
      x0 = 0x0000000000000000

    0x1002f0ba4 <+148>: adrp   x0, -747
      x0 = 0x0000000000000000

    0x1002f0ba8 <+152>: add    x0, x0, #0x4a8            ; =0x4a8 
      x0 = 0x0000000100005000  naot1`__VirtualCall_S_P_Reflection_Core_System_Reflection_Runtime_MethodInfos_RuntimeConstructorInfo__get_RuntimeParameters + 8

    0x1002f0bac <+156>: str    x0, [x29, #0x50]
      x0 = 0x00000001000054a8  naot1`__VirtualCall_S_P_CoreLib_System_Type__get_TypeHandle

    0x1002f0bb0 <+160>: ldr    x0, [x29, #0x58]
      x0 = 0x00000001000054a8  naot1`__VirtualCall_S_P_CoreLib_System_Type__get_TypeHandle

    0x1002f0bb4 <+164>: ldr    x1, [x29, #0x50]
      x0 = 0x0000000000000000

    0x1002f0bb8 <+168>: ldr    wzr, [x0]       <---- EXC_BAD_ACCESS
      x0 = 0x0000000000000000
MichalStrehovsky commented 2 years ago

I think the problem is somewhere in GetTypeFromEETypePtr. The ldr wzr, [x0] line is doing a null check because the call that follows it cannot handle null anymore - it's checking that this for the virtual call is accessible (x0 is supposed to be the this and it was returned by the GetTypeFromEETypePtr call).

https://github.com/dotnet/runtime/blob/2eddb3dd24b0191af2a9d3e0bc6bf25f7cdf63f5/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Type.CoreRT.cs#L25-L46

I expect us to take the SupportsWritableData branch, followed by the !IsAllocated branch, into GetTypeFromEETypePtrSlow.

That one should return a valid object instance (it should be non-null, and if you cast that value to void** and dereference it once, there should be a symbol associated with it (the first field of an object instance is a pointer to its MethodTable (aka vtable). Something is going wrong around there.

am11 commented 2 years ago

It reaches Unsafe.As<T> call on line 35, which seems to be returning null (in X0, assuming it is following standard aarch64 calling convention).

(lldb) c
Process 97360 resuming
Process 97360 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 3.1
    frame #0: 0x000000010014095c naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 108
naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr:
->  0x10014095c <+108>: bl     0x100370490               ; S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<System___Canon>
    0x100140960 <+112>: str    x0, [x29, #0x18]
    0x100140964 <+116>: ldr    x0, [x29, #0x18]
    0x100140968 <+120>: ldp    x29, x30, [sp], #0x60
Target 0: (naot1) stopped.
(lldb) register read
General Purpose Registers:
        x0 = 0x00000001005833f8  __GenericDict_S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<S_P_CoreLib_System_Type>
        x1 = 0x0000000000000000
        x2 = 0x000000000000000a
        x3 = 0x0000000102806668
        x4 = 0x0000000000000020
        x5 = 0x0000000000000003
        x6 = 0x0000000000000001
        x7 = 0x0000000000000000
        x8 = 0x00000001011a11f0
        x9 = 0x000000016fdff218
       x10 = 0x00000001a9734530  libdyld.dylib`tlv_get_addr
       x11 = 0x0000000000000001
       x12 = 0x00000001005fe7a8  g_ephemeral_high
       x13 = 0x0000000102807fd0
       x14 = 0x0000000102806638
       x15 = 0x0000000102806650
       x16 = 0x0000000000000000
       x17 = 0x0000000101304810
       x18 = 0x0000000000000000
       x19 = 0x000000016fdff7a0
       x20 = 0x0000000000000001
       x21 = 0x0000000100000000  naot1`_mh_execute_header
       x22 = 0x0000000000000000
       x23 = 0x0000000000000000
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016fdff200
        lr = 0x000000010014094c  naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 92
        sp = 0x000000016fdff200
        pc = 0x000000010014095c  naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 108
      cpsr = 0x20001000

(lldb) n
Process 97360 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step over
    frame #0: 0x0000000100140960 naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 112
naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr:
->  0x100140960 <+112>: str    x0, [x29, #0x18]
    0x100140964 <+116>: ldr    x0, [x29, #0x18]
    0x100140968 <+120>: ldp    x29, x30, [sp], #0x60
    0x10014096c <+124>: ret    
Target 0: (naot1) stopped.
(lldb) register read
General Purpose Registers:
        x0 = 0x0000000000000000
        x1 = 0x0000000000000000
        x2 = 0x000000000000000a
        x3 = 0x0000000102806668
        x4 = 0x0000000000000020
        x5 = 0x0000000000000003
        x6 = 0x0000000000000001
        x7 = 0x0000000000000000
        x8 = 0x00000001011a11f0
        x9 = 0x000000016fdff218
       x10 = 0x00000001a9734530  libdyld.dylib`tlv_get_addr
       x11 = 0x0000000000000001
       x12 = 0x00000001005fe7a8  g_ephemeral_high
       x13 = 0x0000000102807fd0
       x14 = 0x0000000102806638
       x15 = 0x0000000102806650
       x16 = 0x0000000000000000
       x17 = 0x0000000101304810
       x18 = 0x0000000000000000
       x19 = 0x000000016fdff7a0
       x20 = 0x0000000000000001
       x21 = 0x0000000100000000  naot1`_mh_execute_header
       x22 = 0x0000000000000000
       x23 = 0x0000000000000000
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016fdff200
        lr = 0x0000000100140960  naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 112
        sp = 0x000000016fdff200
        pc = 0x0000000100140960  naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 112
      cpsr = 0x20001000
am11 commented 2 years ago

Didn't knew that in lldb, gui command opens up a TUI 🤩

Frame by frame SS' (bottom up from main to that Unsafe.As call) image image image image image image image image image image

(bit unfortunately that we don't have variables in the upper frames)

MichalStrehovsky commented 2 years ago

Can you try flipping SupportsRelativePointers to false again? I think something is wrong with the reloc from read-only section to read write section.

am11 commented 2 years ago

I tried it before but it wasn't passing withdotnet publish -c Release. ld was failing like:

...
  ld: warning: pointer not aligned at address 0x1003C0482 (___external_NativeReferences_references_End + 56 from obj/release/net7.0/osx-arm64/native/naot1.o)
  ld: warning: pointer not aligned at address 0x1003C047A (___external_NativeReferences_references_End + 48 from obj/release/net7.0/osx-arm64/native/naot1.o)
  ld: warning: pointer not aligned at address 0x1003C0472 (___external_NativeReferences_references_End + 40 from obj/release/net7.0/osx-arm64/native/naot1.o)
  ld: warning: pointer not aligned at address 0x1003C046A (___external_NativeReferences_references_End + 32 from obj/release/net7.0/osx-arm64/native/naot1.o)
  ld: warning: pointer not aligned at address 0x1003C0462 (___external_NativeReferences_references_End + 24 from obj/release/net7.0/osx-arm64/native/naot1.o)
  ld: warning: pointer not aligned at address 0x1003C045A (___external_NativeReferences_references_End + 16 from obj/release/net7.0/osx-arm64/native/naot1.o)
  ld: warning: pointer not aligned at address 0x1003C0452 (___external_NativeReferences_references_End + 8 from obj/release/net7.0/osx-arm64/native/naot1.o)
  ld: warning: pointer not aligned at address 0x1003C044A (___external_NativeReferences_references_End + 0 from obj/release/net7.0/osx-arm64/native/naot1.o)
  ld: unaligned pointer(s) for architecture arm64
... tons of these
  clang: error: linker command failed with exit code 1 (use -v to see invocation)

Now I have tried with Debug, and it builds. Unfortuately lldb does not hit the breakpoints anymore.

This is what I have (note that it still fails in GetTypeFromEETypePtr call):

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00000001a96f99b8 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`__pthread_kill:
->  0x1a96f99b8 <+8>:  b.lo   0x1a96f99d8               ; <+40>
    0x1a96f99bc <+12>: pacibsp 
    0x1a96f99c0 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x1a96f99c4 <+20>: mov    x29, sp
Target 0: (naot1) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00000001a96f99b8 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001a972ceb0 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001a966a314 libsystem_c.dylib`abort + 164
    frame #3: 0x0000000100065ea4 naot1`SystemNative_Abort at pal_threading.c:286:5 [opt]
    frame #4: 0x000000010025eed8 naot1`S_P_CoreLib_Interop_Sys__Abort + 40
    frame #5: 0x000000010013c85c naot1`S_P_CoreLib_System_RuntimeExceptionHelpers__FailFast_1 + 268
    frame #6: 0x000000010013c514 naot1`S_P_CoreLib_System_RuntimeExceptionHelpers__FailFast + 36
    frame #7: 0x00000001001294f4 naot1`S_P_CoreLib_System_Environment__FailFast + 20
    frame #8: 0x00000001000a9cc4 naot1`S_P_TypeLoader_Internal_Runtime_TypeLoader_NativeFormatModuleInfoEnumerator___ctor + 196
    frame #9: 0x00000001000a9bd8 naot1`S_P_TypeLoader_Internal_Runtime_TypeLoader_NativeFormatModuleInfoEnumerable__GetEnumerator + 56
    frame #10: 0x00000001000ce288 naot1`S_P_TypeLoader_Internal_Runtime_TypeLoader_TypeLoaderEnvironment_NamedTypeRuntimeTypeHandleToMetadataHashtable__CreateValueFromKey + 136
    frame #11: 0x000000010029f5c4 naot1`S_P_CoreLib_Internal_TypeSystem_LockFreeReaderHashtable_2<S_P_CoreLib_System_RuntimeTypeHandle__System___Canon>__CreateValueAndEnsureValueIsInTable + 68
    frame #12: 0x000000010029f63c naot1`S_P_CoreLib_Internal_TypeSystem_LockFreeReaderHashtable_2<S_P_CoreLib_System_RuntimeTypeHandle__System___Canon>__GetOrCreateValue + 76
    frame #13: 0x00000001000bef04 naot1`S_P_TypeLoader_Internal_Runtime_TypeLoader_TypeLoaderEnvironment__TryGetMetadataForNamedType + 52
    frame #14: 0x00000001000e52d4 naot1`S_P_Reflection_Execution_Internal_Reflection_Execution_ExecutionEnvironmentImplementation__TryGetMetadataForNamedType + 52
    frame #15: 0x00000001000f02ac naot1`S_P_Reflection_Core_Internal_Reflection_Core_Execution_ExecutionDomain__GetNamedTypeForHandle + 92
    frame #16: 0x00000001000e8bd4 naot1`S_P_Reflection_Execution_Internal_Reflection_Execution_ReflectionExecutionDomainCallbacksImplementation__GetNamedTypeForHandle + 52
    frame #17: 0x0000000100255960 naot1`S_P_CoreLib_Internal_Reflection_Core_NonPortable_RuntimeTypeUnifier__GetRuntimeTypeBypassCache + 288
    frame #18: 0x000000010027fd6c naot1`S_P_CoreLib_Internal_Reflection_Core_NonPortable_RuntimeTypeUnifier_RuntimeTypeHandleToTypeCache__Factory + 60
    frame #19: 0x00000001002c1e4c naot1`S_P_CoreLib_System_Collections_Concurrent_ConcurrentUnifierW_2<IntPtr__System___Canon>__GetOrAdd + 156
    frame #20: 0x000000010025582c naot1`S_P_CoreLib_Internal_Reflection_Core_NonPortable_RuntimeTypeUnifier__GetRuntimeTypeForEEType + 60
    frame #21: 0x000000010013d8d4 naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 164
    frame #22: 0x000000010013d808 naot1`S_P_CoreLib_System_Type__GetTypeFromHandle + 56
    frame #23: 0x00000001002d6c48 naot1`S_P_CoreLib_System_Runtime_Intrinsics_Scalar_1<UIntPtr>__get_AllBitsSet + 72
    frame #24: 0x00000001002d629c naot1`S_P_CoreLib_System_Numerics_Vector_1<UIntPtr>__get_AllBitsSet + 12
    frame #25: 0x000000010034f004 naot1`S_P_CoreLib_System_Numerics_Vector__LessThanOrEqualAll<UInt16> + 36
    frame #26: 0x00000001001e0bbc naot1`S_P_CoreLib_System_Text_ASCIIUtility__GetIndexOfFirstNonAsciiChar_Default + 172
    frame #27: 0x00000001001e0ad8 naot1`S_P_CoreLib_System_Text_ASCIIUtility__GetIndexOfFirstNonAsciiChar + 40
    frame #28: 0x00000001001fdf98 naot1`S_P_CoreLib_System_Text_Unicode_Utf16Utility__GetPointerToFirstInvalidChar + 72
    frame #29: 0x00000001001fa448 naot1`S_P_CoreLib_System_Text_UTF8Encoding__GetByteCountFast + 72
    frame #30: 0x00000001001fa394 naot1`S_P_CoreLib_System_Text_UTF8Encoding__GetByteCountCommon + 84
    frame #31: 0x00000001001fa2c0 naot1`S_P_CoreLib_System_Text_UTF8Encoding__GetByteCount_0 + 160
    frame #32: 0x00000001001e9fb0 naot1`S_P_CoreLib_System_Text_Encoding__GetBytes_2 + 80
    frame #33: 0x0000000100278bc0 naot1`S_P_CoreLib_System_Text_UTF8Encoding_UTF8EncodingSealed__GetBytes + 96
    frame #34: 0x000000010027f424 naot1`S_P_CoreLib_Internal_Console_Error__Write + 68
    frame #35: 0x000000010013c848 naot1`S_P_CoreLib_System_RuntimeExceptionHelpers__FailFast_1 + 248
    frame #36: 0x000000010013c514 naot1`S_P_CoreLib_System_RuntimeExceptionHelpers__FailFast + 36
    frame #37: 0x00000001001294f4 naot1`S_P_CoreLib_System_Environment__FailFast + 20
    frame #38: 0x00000001000a9cc4 naot1`S_P_TypeLoader_Internal_Runtime_TypeLoader_NativeFormatModuleInfoEnumerator___ctor + 196
    frame #39: 0x00000001000a9bd8 naot1`S_P_TypeLoader_Internal_Runtime_TypeLoader_NativeFormatModuleInfoEnumerable__GetEnumerator + 56
    frame #40: 0x00000001000ce288 naot1`S_P_TypeLoader_Internal_Runtime_TypeLoader_TypeLoaderEnvironment_NamedTypeRuntimeTypeHandleToMetadataHashtable__CreateValueFromKey + 136
    frame #41: 0x000000010029f5c4 naot1`S_P_CoreLib_Internal_TypeSystem_LockFreeReaderHashtable_2<S_P_CoreLib_System_RuntimeTypeHandle__System___Canon>__CreateValueAndEnsureValueIsInTable + 68
    frame #42: 0x000000010029f63c naot1`S_P_CoreLib_Internal_TypeSystem_LockFreeReaderHashtable_2<S_P_CoreLib_System_RuntimeTypeHandle__System___Canon>__GetOrCreateValue + 76
    frame #43: 0x00000001000bef04 naot1`S_P_TypeLoader_Internal_Runtime_TypeLoader_TypeLoaderEnvironment__TryGetMetadataForNamedType + 52
    frame #44: 0x00000001000e52d4 naot1`S_P_Reflection_Execution_Internal_Reflection_Execution_ExecutionEnvironmentImplementation__TryGetMetadataForNamedType + 52
    frame #45: 0x00000001000f02ac naot1`S_P_Reflection_Core_Internal_Reflection_Core_Execution_ExecutionDomain__GetNamedTypeForHandle + 92
    frame #46: 0x00000001000e8bd4 naot1`S_P_Reflection_Execution_Internal_Reflection_Execution_ReflectionExecutionDomainCallbacksImplementation__GetNamedTypeForHandle + 52
    frame #47: 0x0000000100255960 naot1`S_P_CoreLib_Internal_Reflection_Core_NonPortable_RuntimeTypeUnifier__GetRuntimeTypeBypassCache + 288
    frame #48: 0x000000010027fd6c naot1`S_P_CoreLib_Internal_Reflection_Core_NonPortable_RuntimeTypeUnifier_RuntimeTypeHandleToTypeCache__Factory + 60
    frame #49: 0x00000001002c1e4c naot1`S_P_CoreLib_System_Collections_Concurrent_ConcurrentUnifierW_2<IntPtr__System___Canon>__GetOrAdd + 156
    frame #50: 0x000000010025582c naot1`S_P_CoreLib_Internal_Reflection_Core_NonPortable_RuntimeTypeUnifier__GetRuntimeTypeForEEType + 60
    frame #51: 0x000000010013d8d4 naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 164
    frame #52: 0x000000010013d808 naot1`S_P_CoreLib_System_Type__GetTypeFromHandle + 56
    frame #53: 0x000000010030b228 naot1`S_P_CoreLib_System_Collections_Generic_Dictionary_2<System___Canon__System___Canon>___ctor_2 + 200
    frame #54: 0x000000010030b11c naot1`S_P_CoreLib_System_Collections_Generic_Dictionary_2<System___Canon__System___Canon>___ctor + 28
    frame #55: 0x0000000100120354 naot1`S_P_CoreLib_System_AppContext__SetData + 116
    frame #56: 0x00000001003149dc naot1`Internal_CompilerGenerated__Module___SetAppContextSwitches + 28
    frame #57: 0x0000000100314afc naot1`__managed__Main + 60
    frame #58: 0x000000010000aea4 naot1`main(argc=1, argv=0x000000016fdff7a0) at main.cpp:205:18 [opt]
    frame #59: 0x0000000100e1d0f4 dyld`start + 520

breakpoints not working with SupportsRelativePointers:false makes it a useless option for me. 😞

I have a feeling that we would need to revisit encodings in methods like EmitLDR() etc. in ARM64Emitter.cs to see if we need to handle values differently for macho (https://github.com/below/HelloSilicon#chapter-4-controlling-programm-flow). Do you have any suggestions about the best was to find out if it is in fact some instruction encoding issue? I was thinking about something like objdump'ing another program like dotnet or corerun and comparing those encodings with what we are emitting in ARM64Emitter.

am11 commented 2 years ago

Seems like this comment: https://github.com/dotnet/runtime/blob/041933fd086f69f839cf38733dd56a1ea9f09866/src/coreclr/nativeaot/System.Private.TypeLoader/src/Internal/Runtime/TypeLoader/CallingConventions.cs#L6 is outdated ever since this commit was merged https://github.com/dotnet/runtime/commit/648437b1b4db6536ae33763b6c9178287c0a52bd. It added osx-arm64 support in the VM, and at the same time refactored the "slot" -based units to bytes/offsets for all architectures. I'll try to update it without unit conversions.

aside - line 26 of CallingConventions.cs seems to have a bug, UNIXAMD64 is not a defined constant in that project (or anywhere else in the repo)?

MichalStrehovsky commented 2 years ago

ld: warning: pointer not aligned at address 0x1003C0482 (___external_NativeReferences_references_End + 56 from obj/release/net7.0/osx-arm64/native/naot1.o)

Should be fixed by #67516.

breakpoints not working with SupportsRelativePointers:false makes it a useless option for me. 😞

That is very odd. SupportsRelativePointers affects very few things in the compiler. I would not expect it to affect debugging.

aside - line 26 of CallingConventions.cs seems to have a bug, UNIXAMD64 is not a defined constant in that project (or anywhere else in the repo)?

CallingConventions.cs is only used in extremely rare circumstances (maybe only when reflection invoking a method with more than 256 parameters). It was written for .NET Native that is Windows only (where it was used more often). You can ignore it.

Going back to the original failure with SupportsRelativePointers=true, can you drill into why GCHandle.IsAllocated returns true? I would expect this to initially be false - the data we're reading from (where the GCHandle is placed) is in the .bss section (that's the special thing about it) and should be zero-initialized. It should not point to an object.

am11 commented 2 years ago

can you drill into why GCHandle.IsAllocated returns true?

bss section is showing up in the app's main module (third-last):

(lldb) image dump sections naot1
Sections for '/Users/am11/projects/naot1/bin/Debug/net7.0/osx-arm64/publish/naot1' (arm64):
  SectID     Type             Load Address                             Perm File Off.  File Size  Flags      Section Name
  ---------- ---------------- ---------------------------------------  ---- ---------- ---------- ---------- ----------------------------
  0x00000100 container        [0x0000000000000000-0x0000000100000000)* ---  0x00000000 0x00000000 0x00000000 naot1.__PAGEZERO
  0x00000200 container        [0x0000000100000000-0x000000010048c000)  r-x  0x00000000 0x0048c000 0x00000000 naot1.__TEXT
  0x00000001 code             [0x0000000100004904-0x000000010007b3e8)  r-x  0x00004904 0x00076ae4 0x80000400 naot1.__TEXT.__text
  0x00000002 code             [0x000000010007b3e8-0x000000010007bd24)  r-x  0x0007b3e8 0x0000093c 0x80000408 naot1.__TEXT.__stubs
  0x00000003 regular          [0x000000010007bd24-0x000000010007bd30)  r-x  0x0007bd24 0x0000000c 0x00000016 naot1.__TEXT.__init_offsets
  0x00000004 regular          [0x000000010007bd30-0x00000001003841e8)  r-x  0x0007bd30 0x003084b8 0x00000000 naot1.__TEXT.__managedcode
  0x00000005 regular          [0x00000001003841f0-0x0000000100486632)  r-x  0x003841f0 0x00102442 0x00000000 naot1.__TEXT.__const
  0x00000006 regular          [0x0000000100486634-0x00000001004884dc)  r-x  0x00486634 0x00001ea8 0x00000000 naot1.__TEXT.__unbox
  0x00000007 data-cstr        [0x00000001004884dc-0x000000010048b753)  r-x  0x004884dc 0x00003277 0x00000002 naot1.__TEXT.__cstring
  0x00000008 compact-unwind   [0x000000010048b754-0x000000010048bb34)  r-x  0x0048b754 0x000003e0 0x00000000 naot1.__TEXT.__unwind_info
  0x00000009 eh-frame         [0x000000010048bb38-0x000000010048bff8)  r-x  0x0048bb38 0x000004c0 0x00000000 naot1.__TEXT.__eh_frame
  0x00000300 container        [0x000000010048c000-0x00000001004cc000)  rw-  0x0048c000 0x00040000 0x00000010 naot1.__DATA_CONST
  0x0000000a data-ptrs        [0x000000010048c000-0x00000001004c76a8)  rw-  0x0048c000 0x0003b6a8 0x00000006 naot1.__DATA_CONST.__got
  0x0000000b regular          [0x00000001004c76a8-0x00000001004c8410)  rw-  0x004c76a8 0x00000d68 0x00000000 naot1.__DATA_CONST.__const
  0x00000400 container        [0x00000001004cc000-0x0000000100614000)  rw-  0x004cc000 0x00138000 0x00000000 naot1.__DATA
  0x0000000c regular          [0x00000001004cc000-0x0000000100557a9a)  rw-  0x004cc000 0x0008ba9a 0x00000000 naot1.__DATA..corert_eh_table
  0x0000000d data             [0x0000000100557b00-0x00000001005fff68)  rw-  0x00557b00 0x000a8468 0x00000000 naot1.__DATA.__data
  0x0000000e regular          [0x00000001005fff68-0x00000001005fff70)  rw-  0x005fff68 0x00000008 0x00000000 naot1.__DATA.__modules
  0x0000000f regular          [0x00000001005fff70-0x0000000100600018)  rw-  0x005fff70 0x000000a8 0x00000013 naot1.__DATA.__thread_vars
  0x00000010 regular          [0x0000000100600018-0x0000000100600100)  rw-  0x00600018 0x000000e8 0x00000011 naot1.__DATA.__thread_data
  0x00000011 regular          [0x0000000100600100-0x0000000100600129)  rw-  0x00000000 0x00000000 0x00000012 naot1.__DATA.__thread_bss
  0x00000012 zero-fill        [0x0000000100600130-0x0000000100609cf8)  rw-  0x00000000 0x00000000 0x00000001 naot1.__DATA.__bss
  0x00000013 zero-fill        [0x0000000100609cf8-0x0000000100610450)  rw-  0x00000000 0x00000000 0x00000001 naot1.__DATA.__common
  0x00000500 container        [0x0000000100614000-0x0000000100f14000)  r--  0x00604000 0x008fca12 0x00000000 naot1.__LINKEDIT

When IsAllocated is visited from GetTypeFromEETypePtr, the state of registers look like this:

(lldb) disas
naot1`S_P_CoreLib_System_Runtime_InteropServices_GCHandle__get_IsAllocated:
->  0x100239720 <+0>:  stp    x29, x30, [sp, #-0x20]!
    0x100239724 <+4>:  mov    x29, sp
    0x100239728 <+8>:  str    x0, [x29, #0x18]
    0x10023972c <+12>: ldr    x0, [x29, #0x18]
    0x100239730 <+16>: ldr    x0, [x0]
    0x100239734 <+20>: mov    w1, wzr
    0x100239738 <+24>: sxtw   x1, w1
    0x10023973c <+28>: cmp    x0, x1
    0x100239740 <+32>: cset   x0, hi
    0x100239744 <+36>: ldp    x29, x30, [sp], #0x20
    0x100239748 <+40>: ret    
    0x10023974c <+44>: udf    #0x0

(lldb) register read
General Purpose Registers:
        x0 = 0x00000001004c3618  (void *)0x0000000100608660: __writableDataString
        x1 = 0x00000001004c3618  (void *)0x0000000100608660: __writableDataString
        x2 = 0x000000000000000a
        x3 = 0x0000000102806668
        x4 = 0x0000000000000020
        x5 = 0x0000000000000003
        x6 = 0x0000000000000001
        x7 = 0x0000000000000000
        x8 = 0x00000001011a11f0
        x9 = 0x000000016fdff218
       x10 = 0x00000001a9734530  libdyld.dylib`tlv_get_addr
       x11 = 0x0000000000000001
       x12 = 0x00000001005fe7a8  g_ephemeral_high
       x13 = 0x0000000102807fd0
       x14 = 0x0000000102806638
       x15 = 0x0000000102806650
       x16 = 0x0000000000000000
       x17 = 0x0000000101304810
       x18 = 0x0000000000000000
       x19 = 0x000000016fdff7a0
       x20 = 0x0000000000000001
       x21 = 0x0000000100000000  naot1`_mh_execute_header
       x22 = 0x0000000000000000
       x23 = 0x0000000000000000
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016fdff200
        lr = 0x0000000100130108  naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 72
        sp = 0x000000016fdff200
        pc = 0x0000000100239720  naot1`S_P_CoreLib_System_Runtime_InteropServices_GCHandle__get_IsAllocated
      cpsr = 0x60001000

when the cursor reaches 0x100239734 <+20>: 0x2a1f03e1 mov w1, wzr, it looks like:

(lldb) n
Process 38694 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step over
    frame #0: 0x0000000100239734 naot1`S_P_CoreLib_System_Runtime_InteropServices_GCHandle__get_IsAllocated + 20
naot1`S_P_CoreLib_System_Runtime_InteropServices_GCHandle__get_IsAllocated:
->  0x100239734 <+20>: mov    w1, wzr
    0x100239738 <+24>: sxtw   x1, w1
    0x10023973c <+28>: cmp    x0, x1
    0x100239740 <+32>: cset   x0, hi
Target 0: (naot1) stopped.
(lldb) register read
General Purpose Registers:
        x0 = 0x0000000100608660  naot1`__writableDataString
        x1 = 0x00000001004c3618  (void *)0x0000000100608660: __writableDataString
        x2 = 0x000000000000000a
        x3 = 0x0000000102806668
        x4 = 0x0000000000000020
        x5 = 0x0000000000000003
        x6 = 0x0000000000000001
        x7 = 0x0000000000000000
        x8 = 0x00000001011a11f0
        x9 = 0x000000016fdff218
       x10 = 0x00000001a9734530  libdyld.dylib`tlv_get_addr
       x11 = 0x0000000000000001
       x12 = 0x00000001005fe7a8  g_ephemeral_high
       x13 = 0x0000000102807fd0
       x14 = 0x0000000102806638
       x15 = 0x0000000102806650
       x16 = 0x0000000000000000
       x17 = 0x0000000101304810
       x18 = 0x0000000000000000
       x19 = 0x000000016fdff7a0
       x20 = 0x0000000000000001
       x21 = 0x0000000100000000  naot1`_mh_execute_header
       x22 = 0x0000000000000000
       x23 = 0x0000000000000000
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016fdff1e0
        lr = 0x0000000100130108  naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 72
        sp = 0x000000016fdff1e0
        pc = 0x0000000100239734  naot1`S_P_CoreLib_System_Runtime_InteropServices_GCHandle__get_IsAllocated + 20
      cpsr = 0x60001000

then x1 becomes zero and the subsequent comparison fails returning true.

MichalStrehovsky commented 2 years ago

Ok, looks like we have a case of a wrong relocation. When we're in GCHandle__get_IsAllocated, this in x0 is 0x00000001004c3618, but from the memory map you posted above, that address is part of naot1.__DATA_CONST.__got. So we're in global offset table, but the code doesn't expect having to dereference the pointer (GOT table is a table of indirections).

So object writer generated a GOT relocation for something that should just store the address of the destination directly. GOT requires an extra dereference to access what the reloc points to.

This might be responsible for the problem: https://github.com/dotnet/llvm-project/pull/185/files#diff-3dd92a728cef8bf36a3e8104cbfcf2b9b901abff46340d48f5cf0a02d3274a2aR455-R456

am11 commented 2 years ago

Trying to figure out how does the RelocationInfoType map to section? e.g. changing ARM64_RELOC_POINTER_TO_GOT to ARM64_RELOC_PAGE21 puts x0 into __DATA.__data. Not sure what makes it place in bss. 🤔

MichalStrehovsky commented 2 years ago

bss is a property of the symbol definition (the symbol that the reloc points to is defined in the bss section), not a property of the reloc to the symbol.

__data might be fine too; I don't know what the linker is doing with this. The important bit is that will be all zeros, the GCHandle will be unallocated and we can write a valid GC handle in that location.

AFAIK ARM64_RELOC_PAGE21 is one of the weirdo relocation types that are to satisfy oddities in ARM instruction encoding. We wouldn't be using them for the data structures.

I'm a bit at loss for where to look next. Normally I would write a piece of C++ that uses the reloc type I'm after and just massage things to emit it, but I don't know if C++ uses 32bit relative pointers for anything, so I can't point you to that. It may well be that the solution will be to do SupportsRelativePointers:false and get that one going instead.

MichalStrehovsky commented 2 years ago

FWIW, the previous iteration (where we ended up with the GOT reloc), looked fine, except for the GOT bit. It was 32 bit, it looked like everything else was pointing to the right thing, but it shouldn't have instructed ld to create GOT entry for it.

am11 commented 2 years ago

bss is a property of the symbol definition (the symbol that the reloc points to is defined in the bss section), not a property of the reloc to the symbol.

Ah, this is correct. I am not sure how/why it is classified as IMAGE_REL_BASED_RELPTR32 to begin with? Is that expected? Note that in addition to these: https://github.com/dotnet/llvm-project/blob/495e374f8fb370a0bf45304fbc835dbf78f2949d/llvm/tools/objwriter/objwriter.cpp#L248-L258, SectionKind.h also has:

  static SectionKind getBSS() { return get(BSS); }
  static SectionKind getBSSLocal() { return get(BSSLocal); }
  static SectionKind getBSSExtern() { return get(BSSExtern); }

if that's something we want to wire up via enum CustomSectionAttributes (in objwriter.h), it is possible.

I have made two changes:

  1. simplified the condition which was added in previous commit am11/llvm-project@495e374f8fb370a0bf45304fbc835dbf78f2949d
  2. rebased my runtime branch against main after a week

with that, although it still fails at the same spot, we get x0 in __DATA.__data section; with ARM64_RELOC_POINTER_TO_GOT:

(lldb) image dump sections naot1
Sections for '/Users/am11/projects/naot1/bin/Debug/net7.0/osx-arm64/publish/naot1' (arm64):
  SectID     Type             Load Address                             Perm File Off.  File Size  Flags      Section Name
  ---------- ---------------- ---------------------------------------  ---- ---------- ---------- ---------- ----------------------------
  0x00000100 container        [0x0000000000000000-0x0000000100000000)* ---  0x00000000 0x00000000 0x00000000 naot1.__PAGEZERO
  0x00000200 container        [0x0000000100000000-0x0000000100488000)  r-x  0x00000000 0x00488000 0x00000000 naot1.__TEXT
  0x00000001 code             [0x0000000100001274-0x0000000100077c78)  r-x  0x00001274 0x00076a04 0x80000400 naot1.__TEXT.__text
  0x00000002 code             [0x0000000100077c78-0x00000001000785b4)  r-x  0x00077c78 0x0000093c 0x80000408 naot1.__TEXT.__stubs
  0x00000003 regular          [0x00000001000785b4-0x00000001000785c0)  r-x  0x000785b4 0x0000000c 0x00000016 naot1.__TEXT.__init_offsets
  0x00000004 regular          [0x00000001000785c0-0x0000000100380a88)  r-x  0x000785c0 0x003084c8 0x00000000 naot1.__TEXT.__managedcode
  0x00000005 regular          [0x0000000100380a90-0x0000000100482642)  r-x  0x00380a90 0x00101bb2 0x00000000 naot1.__TEXT.__const
  0x00000006 regular          [0x0000000100482644-0x00000001004844e4)  r-x  0x00482644 0x00001ea0 0x00000000 naot1.__TEXT.__unbox
  0x00000007 data-cstr        [0x00000001004844e4-0x000000010048775b)  r-x  0x004844e4 0x00003277 0x00000002 naot1.__TEXT.__cstring
  0x00000008 compact-unwind   [0x000000010048775c-0x0000000100487b3c)  r-x  0x0048775c 0x000003e0 0x00000000 naot1.__TEXT.__unwind_info
  0x00000009 eh-frame         [0x0000000100487b40-0x0000000100488000)  r-x  0x00487b40 0x000004c0 0x00000000 naot1.__TEXT.__eh_frame
  0x00000300 container        [0x0000000100488000-0x00000001004c0000)  rw-  0x00488000 0x00038000 0x00000010 naot1.__DATA_CONST
  0x0000000a data-ptrs        [0x0000000100488000-0x00000001004bcc30)  rw-  0x00488000 0x00034c30 0x00000006 naot1.__DATA_CONST.__got
  0x0000000b regular          [0x00000001004bcc30-0x00000001004bd998)  rw-  0x004bcc30 0x00000d68 0x00000000 naot1.__DATA_CONST.__const
  0x00000400 container        [0x00000001004c0000-0x0000000100608000)  rw-  0x004c0000 0x00138000 0x00000000 naot1.__DATA
  0x0000000c regular          [0x00000001004c0000-0x000000010054bb23)  rw-  0x004c0000 0x0008bb23 0x00000000 naot1.__DATA..corert_eh_table
  0x0000000d data             [0x000000010054bc00-0x00000001005f4068)  rw-  0x0054bc00 0x000a8468 0x00000000 naot1.__DATA.__data
  0x0000000e regular          [0x00000001005f4068-0x00000001005f4070)  rw-  0x005f4068 0x00000008 0x00000000 naot1.__DATA.__modules
  0x0000000f regular          [0x00000001005f4070-0x00000001005f4118)  rw-  0x005f4070 0x000000a8 0x00000013 naot1.__DATA.__thread_vars
  0x00000010 regular          [0x00000001005f4118-0x00000001005f4200)  rw-  0x005f4118 0x000000e8 0x00000011 naot1.__DATA.__thread_data
  0x00000011 regular          [0x00000001005f4200-0x00000001005f4229)  rw-  0x00000000 0x00000000 0x00000012 naot1.__DATA.__thread_bss
  0x00000012 zero-fill        [0x00000001005f4230-0x00000001005fddf8)  rw-  0x00000000 0x00000000 0x00000001 naot1.__DATA.__bss
  0x00000013 zero-fill        [0x00000001005fddf8-0x0000000100604550)  rw-  0x00000000 0x00000000 0x00000001 naot1.__DATA.__common
  0x00000500 container        [0x0000000100608000-0x0000000100f04000)  r--  0x005f8000 0x008f8e72 0x00000000 naot1.__LINKEDIT

(lldb) si
Process 46176 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step into
    frame #0: 0x000000010036c060 naot1`S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<System___Canon>
naot1`S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<System___Canon>:
->  0x10036c060 <+0>:  stp    x29, x30, [sp, #-0x20]!
    0x10036c064 <+4>:  mov    x29, sp
    0x10036c068 <+8>:  str    x0, [x29, #0x18]
    0x10036c06c <+12>: str    x1, [x29, #0x10]
Target 0: (naot1) stopped.

(lldb) register read 
General Purpose Registers:
        x0 = 0x0000000100577088  __GenericDict_S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<S_P_CoreLib_System_Type>
        x1 = 0x0000000000000000
        x2 = 0x000000000000000a
        x3 = 0x0000000102806650
        x4 = 0x0000000000000020
        x5 = 0x0000000000000003
        x6 = 0x0000000000000001
        x7 = 0x0000000000000000
        x8 = 0x00000001011911f0
        x9 = 0x000000016fdff218
       x10 = 0x00000001a9734530  libdyld.dylib`tlv_get_addr
       x11 = 0x0000000000000001
       x12 = 0x00000001005f28a8  g_ephemeral_high
       x13 = 0x0000000102807fd0
       x14 = 0x0000000102806620
       x15 = 0x0000000102806638
       x16 = 0x0000000000000000
       x17 = 0x0000000101304630
       x18 = 0x0000000000000000
       x19 = 0x000000016fdff7a0
       x20 = 0x0000000000000001
       x21 = 0x0000000100000000  naot1`_mh_execute_header
       x22 = 0x0000000000000000
       x23 = 0x0000000000000000
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016fdff200
        lr = 0x00000001000ad580  naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 112
        sp = 0x000000016fdff200
        pc = 0x000000010036c060  naot1`S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<System___Canon>
      cpsr = 0x20001000

(lldb) disas
naot1`S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<System___Canon>:
->  0x10036c060 <+0>:  stp    x29, x30, [sp, #-0x20]!
    0x10036c064 <+4>:  mov    x29, sp
    0x10036c068 <+8>:  str    x0, [x29, #0x18]
    0x10036c06c <+12>: str    x1, [x29, #0x10]
    0x10036c070 <+16>: ldr    x0, [x29, #0x10]
    0x10036c074 <+20>: ldp    x29, x30, [sp], #0x20
    0x10036c078 <+24>: ret    
    0x10036c07c <+28>: udf    #0x0

(lldb) c
Process 46176 resuming
Process 46176 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x00000001002e8bf8 naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create + 168
naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create:
->  0x1002e8bf8 <+168>: ldr    wzr, [x0]
    0x1002e8bfc <+172>: blr    x1
    0x1002e8c00 <+176>: str    x0, [x29, #0x48]
    0x1002e8c04 <+180>: ldr    x0, [x29, #0x48]
Target 0: (naot1) stopped.

(lldb) disas
naot1`S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__Create:
    0x1002e8b50 <+0>:   stp    x29, x30, [sp, #-0x90]!
    0x1002e8b54 <+4>:   mov    x29, sp
    0x1002e8b58 <+8>:   add    x9, x29, #0x28            ; =0x28 
    0x1002e8b5c <+12>:  movi.16b v16, #0x0
    0x1002e8b60 <+16>:  stp    q16, q16, [x9]
    0x1002e8b64 <+20>:  stp    q16, q16, [x9, #0x20]
    0x1002e8b68 <+24>:  stp    xzr, xzr, [x9, #0x40]
    0x1002e8b6c <+28>:  str    xzr, [x9, #0x50]
    0x1002e8b70 <+32>:  str    x0, [x29, #0x88]
    0x1002e8b74 <+36>:  str    x0, [x29, #0x80]
    0x1002e8b78 <+40>:  ldr    x0, [x29, #0x80]
    0x1002e8b7c <+44>:  bl     0x100007b74               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_GCStaticBase_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>
    0x1002e8b80 <+48>:  add    x0, x0, #0x8              ; =0x8 
    0x1002e8b84 <+52>:  str    x0, [x29, #0x78]
    0x1002e8b88 <+56>:  ldr    x0, [x29, #0x80]
    0x1002e8b8c <+60>:  bl     0x100008594               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_TypeHandle_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>
    0x1002e8b90 <+64>:  bl     0x1002e8b30               ; S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>__get_SupportsGenericIEquatableInterfaces
    0x1002e8b94 <+68>:  str    w0, [x29, #0x74]
    0x1002e8b98 <+72>:  ldr    x0, [x29, #0x78]
    0x1002e8b9c <+76>:  str    x0, [x29, #0x68]
    0x1002e8ba0 <+80>:  ldr    w0, [x29, #0x74]
    0x1002e8ba4 <+84>:  cbnz   w0, 0x1002e8bc8           ; <+120>
    0x1002e8ba8 <+88>:  nop    
    0x1002e8bac <+92>:  nop    
    0x1002e8bb0 <+96>:  nop    
    0x1002e8bb4 <+100>: nop    
    0x1002e8bb8 <+104>: nop    
    0x1002e8bbc <+108>: ldr    x0, [x29, #0x68]
    0x1002e8bc0 <+112>: str    x0, [x29, #0x28]
    0x1002e8bc4 <+116>: b      0x1002e8bc4               ; <+116>
    0x1002e8bc8 <+120>: ldr    x0, [x29, #0x80]
    0x1002e8bcc <+124>: bl     0x1000085a0               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_TypeHandle_T_System___Canon
    0x1002e8bd0 <+128>: bl     0x1001ce590               ; S_P_CoreLib_Internal_Runtime_CompilerHelpers_LdTokenHelpers__GetRuntimeTypeHandle
    0x1002e8bd4 <+132>: str    x0, [x29, #0x60]
    0x1002e8bd8 <+136>: ldr    x0, [x29, #0x60]
    0x1002e8bdc <+140>: bl     0x1000ad4b0               ; S_P_CoreLib_System_Type__GetTypeFromHandle
    0x1002e8be0 <+144>: str    x0, [x29, #0x58]
    0x1002e8be4 <+148>: adrp   x0, -743
    0x1002e8be8 <+152>: add    x0, x0, #0x538            ; =0x538 
    0x1002e8bec <+156>: str    x0, [x29, #0x50]
    0x1002e8bf0 <+160>: ldr    x0, [x29, #0x58]
    0x1002e8bf4 <+164>: ldr    x1, [x29, #0x50]
->  0x1002e8bf8 <+168>: ldr    wzr, [x0]
    0x1002e8bfc <+172>: blr    x1
    0x1002e8c00 <+176>: str    x0, [x29, #0x48]
    0x1002e8c04 <+180>: ldr    x0, [x29, #0x48]
    0x1002e8c08 <+184>: bl     0x1001c6300               ; S_P_CoreLib_Internal_IntrinsicSupport_EqualityComparerHelpers__GetComparer
    0x1002e8c0c <+188>: str    x0, [x29, #0x40]
    0x1002e8c10 <+192>: ldr    x0, [x29, #0x80]
    0x1002e8c14 <+196>: bl     0x10000926c               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_MethodDictionary_S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>>
    0x1002e8c18 <+200>: str    x0, [x29, #0x20]
    0x1002e8c1c <+204>: ldr    x0, [x29, #0x20]
    0x1002e8c20 <+208>: ldr    x1, [x29, #0x40]
    0x1002e8c24 <+212>: bl     0x10036c060               ; S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<System___Canon>
    0x1002e8c28 <+216>: str    x0, [x29, #0x38]
    0x1002e8c2c <+220>: ldr    x0, [x29, #0x80]
    0x1002e8c30 <+224>: bl     0x100009260               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_MethodDictionary_S_P_CoreLib_System_Threading_Interlocked__CompareExchange_3<S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>>
    0x1002e8c34 <+228>: str    x0, [x29, #0x18]
    0x1002e8c38 <+232>: ldr    x0, [x29, #0x18]
    0x1002e8c3c <+236>: ldr    x1, [x29, #0x68]
    0x1002e8c40 <+240>: ldr    x2, [x29, #0x38]
    0x1002e8c44 <+244>: mov    x3, xzr
    0x1002e8c48 <+248>: bl     0x10036b340               ; S_P_CoreLib_System_Threading_Interlocked__CompareExchange_3<System___Canon>
    0x1002e8c4c <+252>: str    x0, [x29, #0x30]
    0x1002e8c50 <+256>: nop    
    0x1002e8c54 <+260>: ldr    x0, [x29, #0x80]
    0x1002e8c58 <+264>: bl     0x100007b74               ; __GenericLookupFromType_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<System___Canon>_GCStaticBase_S_P_CoreLib_System_Collections_Generic_EqualityComparer_1<T_System___Canon>
    0x1002e8c5c <+268>: ldr    x0, [x0, #0x8]
    0x1002e8c60 <+272>: ldp    x29, x30, [sp], #0x90
    0x1002e8c64 <+276>: ret    
    0x1002e8c68 <+280>: udf    #0x0
    0x1002e8c6c <+284>: udf    #0x0

These changes in main might have something to do with this slight improvement: https://github.com/dotnet/runtime/commit/4019e83878a81465f6e42e8502b53bc5d1752f81 and https://github.com/dotnet/runtime/commit/64096424f2c61453819522450945388734825bea.

MichalStrehovsky commented 2 years ago

Ah, this is correct. I am not sure how/why it is classified as IMAGE_REL_BASED_RELPTR32 to begin with? Is that expected?

The problematic reloc around the GCHandle code is generated as such here: https://github.com/dotnet/runtime/blob/34f2b587f3512d3974d73b316e565194df34eea3/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/EETypeNode.cs#L925-L938

We use IMAGE_REL_BASED_RELPTR32 because the target of the reloc is within the same image and we don't want to waste a full pointer on it. It's not the only place that uses IMAGE_REL_BASED_RELPTR32 for this reason. SupportsRelativePointers guards the use of these relocs in data structures because there's targets like WASM that cannot express such relocation.

You could rule out whether the problem is related to BSS by changing the section here to data: https://github.com/dotnet/runtime/blob/938f2e3105ce9a851fedac96a210ffabaccb7dc3/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs#L241. It will emit explicit zeros into the image instead of relying on BSS.

These changes in main might have something to do with this slight improvement: https://github.com/dotnet/runtime/commit/4019e83878a81465f6e42e8502b53bc5d1752f81 and https://github.com/dotnet/runtime/commit/64096424f2c61453819522450945388734825bea.

None of the files in those commits compile as part of NativeAOT so its unlikely.

The null dereference you're seeing is still caused by the code around the GCHandle thinking that there's a GC handle with some null object.

The GCHandle in question is 8 bytes of what's supposed to be the BSS section.

am11 commented 2 years ago

Changing Bss to Data in that BlobNode ctor gives:

(lldb) register read 
General Purpose Registers:
        x0 = 0x000000010056f088  __GenericDict_S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<S_P_CoreLib_System_Type>
        x1 = 0x00000001020075e8
        x2 = 0x000000000000000a
        x3 = 0x0000000101404080
        x4 = 0x0000000000000061
        x5 = 0x000000016fdfee58
        x6 = 0x0000000000000000
        x7 = 0x0000000000000000
        x8 = 0x00000001012911f0
        x9 = 0x000000016fdff218
       x10 = 0x00000001a9734530  libdyld.dylib`tlv_get_addr
       x11 = 0x0000000000000001
       x12 = 0x00000001005f5e60  naot1`g_highest_address
       x13 = 0x0000000102007fd0
       x14 = 0x000000016fdff320
       x15 = 0x0000000102006460
       x16 = 0x0000000000000000
       x17 = 0x0000000101404080
       x18 = 0x0000000000000000
       x19 = 0x000000016fdff7a0
       x20 = 0x0000000000000001
       x21 = 0x0000000100000000  naot1`_mh_execute_header
       x22 = 0x0000000000000000
       x23 = 0x0000000000000000
       x24 = 0x0000000000000000
       x25 = 0x0000000000000000
       x26 = 0x0000000000000000
       x27 = 0x0000000000000000
       x28 = 0x0000000000000000
        fp = 0x000000016fdff200
        lr = 0x00000001000ad580  naot1`S_P_CoreLib_System_Type__GetTypeFromEETypePtr + 112
        sp = 0x000000016fdff200
        pc = 0x000000010036c060  naot1`S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<System___Canon>
      cpsr = 0x20001000

it is __data section:

  0x0000000d data             [0x0000000100543c00-0x00000001005f4f68)  rw-  0x00543c00 0x000b1368 0x00000000 naot1.__DATA.__data

and the As<T> call in S_P_CoreLib_System_Type__GetTypeFromEETypePtr does not fail (it later gives EXC_BAD_ACCESS from S_P_CoreLib_Internal_Runtime_IatAwarePointer_1<S_P_CoreLib_Internal_Runtime_MethodTable>__get_Value).

Note that while x0 still has the same value (__GenericDict_S_P_CoreLib_System_Runtime_CompilerServices_Unsafe__As<S_P_CoreLib_System_Type>), the difference between previously failing call to As<T> and this successful execution is address of x1; before it was null, now it has the address 0x00000001020075e8, which is outside of the entire module address ranges (0x0000000000000000 - 0x0000000100ef8000).

MichalStrehovsky commented 2 years ago

Yeah, the reloc looks still wrong then.

In your objwriter change, what is forcing the RelocType = unsigned(MachO::ARM64_RELOC_POINTER_TO_GOT); part around FK_PCRel_4? Is anything asserting/crashing if that line is not there? A thing with GOT in the name doesn't sound right.

am11 commented 2 years ago

Removing RelocType = unsigned(MachO::ARM64_RELOC_POINTER_TO_GOT); line from AArch64MachObjectWriter.cpp takes us back to the error from the very top post:

...
  naot1 -> /Users/am11/projects/naot1/bin/Debug/net7.0/osx-arm64/naot1.dll
  Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeCoreRT
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: fixup value out of range
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: fixup value out of range
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: fixup value out of range
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: fixup value out of range
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: fixup value out of range
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: fixup value out of range
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: fixup value out of range
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: fixup value out of range
  <unknown>:0: error: ADR/ADRP relocations must be GOT relative
...
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  <unknown>:0: error: unknown AArch64 fixup kind!
  ld: malformed __LD,__compact_unwind section, bad length file 'obj/Debug/net7.0/osx-arm64/native/naot1.o'
  clang: error: linker command failed with exit code 1 (use -v to see invocation)

(although useCompactUnwind still returns false per the change we made earlier)

MichalStrehovsky commented 2 years ago

I don't understand how deleting the RelocType = unsigned(MachO::ARM64_RELOC_POINTER_TO_GOT); line gets us to the ADR/ADRP relocations must be GOT relative message - the message comes from a different switch block in the same method - how does deleting the single line gets us to a different switch block?

am11 commented 2 years ago

I haven't stepped through that code, but it could be recordRelocation is called again for same symbol with immediate flag set? I don't have any other local changes.

MichalStrehovsky commented 2 years ago

Are you 100% sure you're using the objwriter you built? I crosscompiled from windows, but I don't see those errors printed even if I remove the line in question (just the one line, not the entire case).

am11 commented 2 years ago

Indeed case of bad caching. 😞 I tried again from clean state (takes 13-15 minutes to clean the nuget caches and build both repos on M1 pro). The error is there, but it is different:

  Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeCoreRT
  ld: in section __DATA,.corert_eh_table reloc 0: pcrel and ARM64_RELOC_UNSIGNED not supported file 'obj/Debug/net7.0/osx-arm64/native/naot1.o'
  clang: error: linker command failed with exit code 1 (use -v to see invocation)

Uncommenting that (one) RelocType line fixes this error and publishes the app which I was debugging. 

  Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeCoreRT
  ld: warning: can't parse dwarf compilation unit info in obj/Debug/net7.0/osx-arm64/native/naot1.o
  naot1 -> /Users/am11/projects/naot1/bin/Debug/net7.0/osx-arm64/publish

Note that earlier we were getting type 15 error from corert_eh_table (https://github.com/dotnet/runtime/issues/67232#issuecomment-1083078234). Looks like case FK_PCRel_4 needs a rework.

MichalStrehovsky commented 2 years ago

Looking at the relocations that seem to be actually supported around the error message in https://github.com/zzjconcent/ollvm/blob/d28107a3d30dfadde47ae0118d62d8d7951491f9/src/lib/ExecutionEngine/RuntimeDyld/Targets/RuntimeDyldMachOAArch64.h doesn't give us many options. It might be possible to represent this as ARM64_RELOC_SUBTRACTOR (make a temp symbol for current address and do a difference to destination). That's what PC relative is about anyway.

Or the other option is to take the !SupportsRelativePointers path because that's clearly where Apple wants us to go after they gimped their platform again and removed a reloc kind that exists everywhere else.

am11 commented 2 years ago

With SupportRelativePointers=false, now backtraces are also working. We reach: https://github.com/dotnet/runtime/blob/6de7147b9266d7730b0d73ba67632b0c198cb11e/src/coreclr/nativeaot/System.Private.TypeLoader/src/Internal/Runtime/TypeLoader/ModuleList.cs#L367 and it returns false followed by the FailFast from next line which results in SIGABRT. This Find method is returning null (so it doesn't enter if block): https://github.com/dotnet/runtime/blob/6de7147b9266d7730b0d73ba67632b0c198cb11e/src/coreclr/nativeaot/Common/src/System/Collections/Generic/LowLevelDictionary.cs#L79-L80

same callstack as https://github.com/dotnet/runtime/issues/67232#issuecomment-1086634886.

am11 commented 2 years ago

According to x/64s $x0, the message is:

Invalid module requested in enumeration: 69446c657665c77

This is a huge address out of all module ranges. 🤔

MichalStrehovsky commented 2 years ago

@am11 - thanks for your help with this so far! I haven't forgotten about this, but my son was born and other things are now taking priority. I don't expect things to be normal for the next few weeks.