dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.42k stars 4.76k forks source link

[Hot Reload] metadata.c:1326, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met #93860

Closed rolfbjarne closed 10 months ago

rolfbjarne commented 1 year ago

From @jeromelaban on Fri, 20 Oct 2023 03:53:31 GMT

Steps to Reproduce

  1. Create a maui app (net8 rc2)
  2. Add a [CreateNewOnMetadataUpdate] attribute on MainPage
  3. Build the app
  4. Run the app
  5. Add a ToString(); call in the ctor, or any other MainPage modification
  6. Perform a C# hot reload

Expected Behavior

The code is updated without crashing.

Actual Behavior

Sometimes:

error: * Assertion at /Users/runner/work/1/s/src/mono/mono/metadata/metadata.c:1326, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

Sometimes:

=================================================================
    Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
    Native stacktrace:
=================================================================
    0x104aba1d5 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info
    0x104a58b6e - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_handle_native_crash
    0x1049abaff - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_sigsegv_signal_handler_debug
    0x7ff8377405ed - /usr/lib/system/libsystem_platform.dylib : _sigtramp
    0x0 - Unknown
    0x103b33fd6 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : get_types_for_source_file
    0x103b3ce1d - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : monoeg_g_hash_table_foreach
    0x103b25917 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : mono_process_dbg_packet
    0x103b2d3ff - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : mono_debugger_agent_receive_and_process_command
    0x103b300f0 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : debugger_thread
    0x104bbb4fe - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : start_wrapper
    0x7ff83774d1d3 - /usr/lib/system/libsystem_pthread.dylib : _pthread_start
    0x7ff837748bd3 - /usr/lib/system/libsystem_pthread.dylib : thread_start

Environment

Installed Workload Id      Manifest Version                     Installation Source
----------------------------------------------------------------------------------------------------------------------
maui-ios                   8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444
ios                        16.4.8968-net8-rc2/8.0.100-rc.2      VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maccatalyst                16.4.8968-net8-rc2/8.0.100-rc.2      VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maui-android               8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444
android                    34.0.0-rc.2.468/8.0.100-rc.2         VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maui-windows               8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27
maui-maccatalyst           8.0.0-rc.2.9373/8.0.100-rc.2         VS 17.7.34009.444

Build Logs

Available on demand.

Copied from original issue xamarin/xamarin-macios#19277

rolfbjarne commented 1 year ago

From @rolfbjarne on Fri, 20 Oct 2023 08:44:49 GMT

@jeromelaban this looks like a crash in the debugger; does the app run if you don't attach a debugger (just tap on it)?

rolfbjarne commented 1 year ago

From @jeromelaban on Fri, 20 Oct 2023 11:30:28 GMT

@rolfbjarne thanks for looking into it! No, the app does not crash, but there's no hot reload that can happen either. The issue only happens when explicitly triggering a hot reload (which happens through the debugger in this case).

It may be a runtime issue, though, since it fails in a very similar way under Android.

I can open a runtime issue if you think it's more appropriate.

rolfbjarne commented 1 year ago

From @jeromelaban on Fri, 20 Oct 2023 12:36:24 GMT

@rolfbjarne I just discovered that I missed some steps in the original description, my apologies. Hot reloading is required for this to fail.

rolfbjarne commented 1 year ago

From @rolfbjarne on Mon, 23 Oct 2023 12:15:07 GMT

I can open a runtime issue if you think it's more appropriate.

Yes, this looks like a runtime issue, so the dotnet/runtime repository would be a better place for it.

I'll move it there.

jeromelaban commented 1 year ago

@rolfbjarne I was under the impression that using System.Reflection.Metadata.MetadataUpdater.ApplyUpdate was more stable, but it's still failing with this same error (when compared to the debugger doing the same through its own APIs), sometimes immediately, sometimes after a few updates.

jeromelaban commented 1 year ago

Also, the issue happens on webassembly, which makes sense as it's mono underneath. (/cc @lewing)

lambdageek commented 1 year ago

CreateNewOnMetadataUpdate is hard to use (the old class name is still around and valid) and I don't think Maui has been updated to use it (although I could be wrong). But the runtime crash is not expected.

lambdageek commented 1 year ago

@jeromelaban do you have separate repro steps for Wasm? I think it's probably crashing in a different place for the same underlying reason. It would be helpful to see the other stack trace

jeromelaban commented 1 year ago

@lambdageek I'll try to build one using blazor.

In the meantime it really is almost the same place, probably in slight version differences:

[MONO] * Assertion at /__w/runtime/src/mono/mono/metadata/metadata.c:1321, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

    at mono_wasm_stringify_as_error_with_stack (dotnet.js:5:620)
    at Object.mono_wasm_trace_logger (dotnet.js:5:985)
    at _mono_wasm_trace_logger (dotnet.js:14:111110)
    at dotnet.wasm:0x9f18
    at dotnet.wasm:0xabc71
    at dotnet.wasm:0x1c8ae2
    at dotnet.wasm:0x1c8b41
    at dotnet.wasm:0x1c8b75
    at dotnet.wasm:0x822de
    at dotnet.wasm:0x821ff
jeromelaban commented 1 year ago

I'm still not able to get a repro with blazor, but here's another log, which includes the call to dump_update_summary: 93860-crash_trace.txt

Here's an excerpt of the log:

[MONO] no implementation for interface method testhr01.MainPage#1.IMainPage_Bindings::Initialize() in class testhr01.MainPage#1/MainPage_Bindings
[MONO] METHOD get_Owner()
[MONO] METHOD set_Owner(testhr01.MainPage#1)
[MONO] METHOD .ctor(testhr01.MainPage#1)
[MONO] METHOD testhr01.MainPage.IMainPage_Bindings.Initialize()
[MONO] METHOD testhr01.MainPage.IMainPage_Bindings.Update()
[MONO] METHOD testhr01.MainPage.IMainPage_Bindings.UpdateResources()
[MONO] METHOD testhr01.MainPage.IMainPage_Bindings.StopTracking()
[MONO] Running class .cctor for System.SR from 'System.Private.CoreLib.dll'
[MONO] Running class .cctor for System.Resources.ResourceManager from 'System.Private.CoreLib.dll'
[MONO] Running class .cctor for System.Resources.ResourceReader from 'System.Private.CoreLib.dll'
[MONO] Running class .cctor for System.Resources.FastResourceComparer from 'System.Private.CoreLib.dll'
Error: [MONO] * Assertion at /__w/Uno.DotnetRuntime.WebAssembly/Uno.DotnetRuntime.WebAssembly/runtime/src/mono/mono/metadata/metadata.c:1321, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

Update: Still trying to get a repro, and it seems to be related to nested types marked as CreateNewOnMetadataUpdate. Will update as I find more.

lambdageek commented 1 year ago

@jeromelaban unfortunately this part ofhte trace:

[MONO] * Assertion at /__w/runtime/src/mono/mono/metadata/metadata.c:1321, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

    at mono_wasm_stringify_as_error_with_stack (dotnet.js:5:620)
    at Object.mono_wasm_trace_logger (dotnet.js:5:985)
    at _mono_wasm_trace_logger (dotnet.js:14:111110)

is not useful. that's just the part where the runtime is reporting that there was a failed assertion.

what I really need is the symbolic names for this part:

 at dotnet.wasm:0x9f18
    at dotnet.wasm:0xabc71
    at dotnet.wasm:0x1c8ae2
    at dotnet.wasm:0x1c8b41
    at dotnet.wasm:0x1c8b75
    at dotnet.wasm:0x822de
    at dotnet.wasm:0x821ff

The line in metadata.c:1321 is in mono_metadata_decode_row_raw which is called from from around 200 places.

On ios it is clearly something in the debugger. But on wasm I wouldn't expect the updates to come in via the debugger (and I don't think the debugger is active right - you're just running dotnet-watch?) so it's some other place.

@lewing @pavelsavara do we have some way so symbolicate wasm crashes?

ghost commented 1 year ago

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Issue Details
_From @jeromelaban on Fri, 20 Oct 2023 03:53:31 GMT_ ### Steps to Reproduce 1. Create a maui app (net8 rc2) 2. Add a `[CreateNewOnMetadataUpdate]` attribute on `MainPage` 3. Build the app 4. Run the app 5. Add a `ToString();` call in the ctor, or any other `MainPage` modification 6. Perform a C# hot reload ### Expected Behavior The code is updated without crashing. ### Actual Behavior Sometimes: ``` error: * Assertion at /Users/runner/work/1/s/src/mono/mono/metadata/metadata.c:1326, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met ``` Sometimes: ``` ================================================================= Native Crash Reporting ================================================================= Got a SIGSEGV while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. ================================================================= ================================================================= Native stacktrace: ================================================================= 0x104aba1d5 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info 0x104a58b6e - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_handle_native_crash 0x1049abaff - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : mono_sigsegv_signal_handler_debug 0x7ff8377405ed - /usr/lib/system/libsystem_platform.dylib : _sigtramp 0x0 - Unknown 0x103b33fd6 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : get_types_for_source_file 0x103b3ce1d - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : monoeg_g_hash_table_foreach 0x103b25917 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : mono_process_dbg_packet 0x103b2d3ff - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : mono_debugger_agent_receive_and_process_command 0x103b300f0 - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmono-component-debugger.dylib : debugger_thread 0x104bbb4fe - /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/FD06AA1D-DB01-4BC1-B692-B59CC36FB231/MauiApp3.app/libmonosgen-2.0.dylib : start_wrapper 0x7ff83774d1d3 - /usr/lib/system/libsystem_pthread.dylib : _pthread_start 0x7ff837748bd3 - /usr/lib/system/libsystem_pthread.dylib : thread_start ``` ### Environment - VS 2022 17.8 Preview 4 - .NET 8.0.100-rc.2.23502.2 ``` Installed Workload Id Manifest Version Installation Source ---------------------------------------------------------------------------------------------------------------------- maui-ios 8.0.0-rc.2.9373/8.0.100-rc.2 VS 17.7.34009.444 ios 16.4.8968-net8-rc2/8.0.100-rc.2 VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27 maccatalyst 16.4.8968-net8-rc2/8.0.100-rc.2 VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27 maui-android 8.0.0-rc.2.9373/8.0.100-rc.2 VS 17.7.34009.444 android 34.0.0-rc.2.468/8.0.100-rc.2 VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27 maui-windows 8.0.0-rc.2.9373/8.0.100-rc.2 VS 17.7.34009.444, VS 17.8.34212.112, VS 17.8.34112.27 maui-maccatalyst 8.0.0-rc.2.9373/8.0.100-rc.2 VS 17.7.34009.444 ``` ### Build Logs Available on demand. _Copied from original issue xamarin/xamarin-macios#19277_
Author: rolfbjarne
Assignees: lambdageek
Labels: `arch-wasm`, `os-ios`, `area-EnC-mono`
Milestone: 8.0.x
lambdageek commented 1 year ago

I was under the impression that using System.Reflection.Metadata.MetadataUpdater.ApplyUpdate was more stable

This is not the case, btw. The debugger and the managed API call down into identical code. The only way in which not using the debugger is more stable is that there is less code observing the hot reload changes. (The debugger does things like query the names of locals, get all the members of a class, etc etc, that typical code could do with reflection, but is pretty rare in general).

jeromelaban commented 1 year ago

Thanks for the updates @lambdageek.

is not useful. that's just the part where the runtime is reporting that there was a failed assertion.

Agreed it's not useful at this point, I'm trying to get a repro for blazor specifically and have symbols for that trace as well, no luck so far (having both the crash and symbols enabled).

But on wasm I wouldn't expect the updates to come in via the debugger (and I don't think the debugger is active right - you're just running dotnet-watch?) so it's some other place.

For wasm it's not using the debugger, indeed, the browserlink feature is used to provide the updates.

Aside from that, on the iOS front from my own Uno tests, I enable traces for everything, and here's what I got: 93860-crash_trace_ios.txt

2023-10-24 09:40:15.521066-0400 testhr04.Mobile[94697:35244945] info: >>> EnC delta for base=/Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll (generation 2) applied
2023-10-24 09:40:15.705544-0400 testhr04.Mobile[94697:35244905] info: NO Found interfaces for class 0x0200000f
2023-10-24 09:40:15.705696-0400 testhr04.Mobile[94697:35244905] info: Creating class '.IMainPage_Bindings' from skeleton (first_method_idx = 0x00000053, count = 0x00000004, first_field_idx = 0x00000000, count=0x00000000)
2023-10-24 09:40:15.705888-0400 testhr04.Mobile[94697:35244905] debug: Requesting loading reference 18 (of 25) of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll
2023-10-24 09:40:15.706043-0400 testhr04.Mobile[94697:35244905] debug: Loading reference 18 of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll (default ALC), looking for System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
2023-10-24 09:40:15.706245-0400 testhr04.Mobile[94697:35244905] debug: Request to load System.Runtime in alc 0x600003695300
2023-10-24 09:40:15.706378-0400 testhr04.Mobile[94697:35244905] debug: Assembly already loaded in the active ALC: 'System.Runtime'.
2023-10-24 09:40:15.706493-0400 testhr04.Mobile[94697:35244905] debug: Assembly Ref addref testhr04[0x60000389ce10] -> System.Runtime[0x600003884000]: 107
2023-10-24 09:40:15.706624-0400 testhr04.Mobile[94697:35244905] info: Found interfaces for class 0x02000010 starting at 0x00000003
2023-10-24 09:40:15.706757-0400 testhr04.Mobile[94697:35244905] info: Creating class '.MainPage_Bindings' from skeleton (first_method_idx = 0x00000057, count = 0x00000007, first_field_idx = 0x00000032, count=0x00000001)
2023-10-24 09:40:15.706860-0400 testhr04.Mobile[94697:35244905] info: NO Found interfaces for class 0x02000011
2023-10-24 09:40:15.706984-0400 testhr04.Mobile[94697:35244905] info: Creating class '.<>c' from skeleton (first_method_idx = 0x0000005e, count = 0x00000007, first_field_idx = 0x00000033, count=0x00000006)
2023-10-24 09:40:15.708790-0400 testhr04.Mobile[94697:35244905] debug: Requesting loading reference 19 (of 25) of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll
2023-10-24 09:40:15.708885-0400 testhr04.Mobile[94697:35244905] debug: Loading reference 19 of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll (default ALC), looking for System.Runtime.Loader, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
2023-10-24 09:40:15.709068-0400 testhr04.Mobile[94697:35244905] debug: Request to load System.Runtime.Loader in alc 0x600003695300
2023-10-24 09:40:15.709230-0400 testhr04.Mobile[94697:35244905] debug: Assembly already loaded in the active ALC: 'System.Runtime.Loader'.
2023-10-24 09:40:15.709352-0400 testhr04.Mobile[94697:35244905] debug: Assembly Ref addref testhr04[0x60000389ce10] -> System.Runtime.Loader[0x60000389c1b0]: 8
2023-10-24 09:40:15.709485-0400 testhr04.Mobile[94697:35244905] debug: Requesting loading reference 20 (of 25) of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll
2023-10-24 09:40:15.709620-0400 testhr04.Mobile[94697:35244905] debug: Loading reference 20 of /Users/jay2/Library/Developer/CoreSimulator/Devices/F4A03A05-42EF-47DF-8C0B-0DB5B0E49D76/data/Containers/Bundle/Application/9BA8BC6D-7CA1-4E02-9D18-107DE14D9B72/testhr04.Mobile.app/testhr04.dll (default ALC), looking for Microsoft.iOS, Version=16.4.0.0, Culture=neutral, PublicKeyToken=84e04ff9cfb79065
2023-10-24 09:40:15.709741-0400 testhr04.Mobile[94697:35244905] debug: Request to load Microsoft.iOS in alc 0x600003695300
2023-10-24 09:40:15.709862-0400 testhr04.Mobile[94697:35244905] debug: Assembly already loaded in the active ALC: 'Microsoft.iOS'.
2023-10-24 09:40:15.709957-0400 testhr04.Mobile[94697:35244905] debug: Assembly Ref addref testhr04[0x60000389ce10] -> Microsoft.iOS[0x60000389c120]: 17
2023-10-24 09:40:16.318353-0400 testhr04.Mobile[94697:35244905] info: NO Found interfaces for class 0x0200000e
2023-10-24 09:40:16.318669-0400 testhr04.Mobile[94697:35244905] debug: method lookup idx=0x0000004c returned gen=2 il=0x7fc15cdbed2c
2023-10-24 09:40:16.318839-0400 testhr04.Mobile[94697:35244905] debug: method lookup idx=0x0000004c returned gen=2 il=0x600004828530
2023-10-24 09:40:16.319023-0400 testhr04.Mobile[94697:35244905] debug: member_parent lookup: 0x0400002f returned 0x0200000e
2023-10-24 09:40:16.319194-0400 testhr04.Mobile[94697:35244905] debug: member_parent lookup: 0x04000030 returned 0x0200000e
2023-10-24 09:40:16.333531-0400 testhr04.Mobile[94697:35244905] error: * Assertion at /Users/runner/work/1/s/src/mono/mono/metadata/metadata.c:1326, condition `GINT_TO_UINT32(idx) < table_info_get_rows (t)' not met

That being said, the crash on MAUI is somewhat different (using the repro steps at the top): 93860-crash_trace_ios_maui.txt, so maybe it's two separate issues, after all.

jeromelaban commented 1 year ago

@lambdageek I've been trying to get to a repro for the metadata.c issue inside a MAUI app, but I can't get the same conditions that fail. Here's a repro that uses Uno Platform and vscode on mac to repro the crash for iOS:

To repro on macOS:

Notice that the app will crash with the metadata.c:1326 assertion.

The app has all runtime debugging enabled for easier troubleshooting.

lambdageek commented 1 year ago

@jeromelaban I can repro the Uno crash. It's always on the second update for me.

crash reporter shows this stack:

8   libsystem_c.dylib                      0x1241ecd60 abort + 133
9   libxamarin-dotnet-debug.dylib          0x10ae01340 log_callback(char const*, char const*, char const*, int, void*) + 64
10  libmonosgen-2.0.dylib                  0x10b508632 monoeg_g_logv_nofree + 194
11  libmonosgen-2.0.dylib                  0x10b5087af monoeg_assertion_message + 143
12  libmonosgen-2.0.dylib                  0x10b5087ea mono_assertion_message + 26
13  libmonosgen-2.0.dylib                  0x10b58d364 mono_metadata_decode_row + 436
14  libmonosgen-2.0.dylib                  0x10b552801 mono_ppdb_lookup_location_internal + 65
15  libmonosgen-2.0.dylib                  0x10b59c6d8 mono_debug_lookup_source_location + 312
16  libmonosgen-2.0.dylib                  0x10b4548ca mono_get_trace + 698
17  libmonosgen-2.0.dylib                  0x10b566a8b ves_icall_System_Diagnostics_StackTrace_GetTrace + 43
18  libmonosgen-2.0.dylib                  0x10b4cdde8 do_icall + 248
19  libmonosgen-2.0.dylib                  0x10b4cc60e do_icall_wrapper + 350
20  libmonosgen-2.0.dylib                  0x10b4bd2f6 mono_interp_exec_method + 3990
21  libmonosgen-2.0.dylib                  0x10b4ce1e5 interp_entry + 453
22  libmonosgen-2.0.dylib                  0x10b4cea17 interp_entry_static_1 + 55

which looks like there was already some fault and then the StackTrace.GetTrace method was invoked and it crashed in the ppdb infrastructure.

I'm going to try to make some custom builds of the runtime for iossimulator with extra logging.

by the way, is it possible to build the project for iossimulator-arm64. My naive attempts failed (the build tried to link some SkiaSharp native library that was built for ios not iossimulator and at that point I gave up). Not a big deal (I can make iossimulator-x64 custom runtimes, so I'm not blocked), just curious.

jeromelaban commented 1 year ago

@lambdageek it is supposed to be supported, that's what SkiaSharp 2.88.6 was about, but I'll try on arm64 and let you know if there's something missing (could be the native assets for iOS that are incorrect). We could also completely remove all skiasharp references, they're not needed for this issue, but I'll post an updated sample without it so you don't have to fiddle.

jeromelaban commented 1 year ago

@lambdageek I was not able to fail the build with the original sample on arm64, but here's an updated one without Skia dependencies: issue83860-2.zip.

jeromelaban commented 1 year ago

@lambdageek Furthermore, seeing the stacktrace that you posted got me thinking that the interpreter could play a role, but it's not, even when disabling it.

Adding:

<MtouchExtraArgs>$(MtouchExtraArgs) --setenv=DOTNET_MODIFIABLE_ASSEMBLIES=debug</MtouchExtraArgs>

(because there's no interpreter) does not help and the failure in metadata.c is the same.

lambdageek commented 1 year ago

I'm back to spending some time on this.

Update actually... Mono might be broken here even if the hot reload delta comes in with a PDB delta - I think if a new method throws we might not be getting the right source info in mono_debug_lookup_source_location no matter what.

I think one problem might be that when Uno sends the hot reload delta, you're not sending the PDB delta, even though the baseline assembly had a PDB. And I'm pretty sure Mono is not expecting that. (I'm going to try to make Mono more resilient to this case)

I'm not sure what further issue that will uncover - it looks like the runtime was trying to generate a stack trace for some other exception

jeromelaban commented 1 year ago

@lambdageek Interesting! So at this time, Uno is effectively sending the PDBs (at least the field for the PDB delta is not empty), so I'm wondering if it's because the PDBs is not generated by roslyn at all? The issue is manifesting itself the same way on VS Windows, so would it mean that VS Windows and/or Roslyn has the same incorrect behavior? (Uno is not managing the deltas on VS Windows, it's all done by VS)

lambdageek commented 1 year ago

@jeromelaban I was wrong. Even if Uno (or VS Win) is sending PDB deltas, the code in mono_debug_lookup_source_location just doesn't handle the case of newly added methods correctly.

A simple repro is

$ dotnet new blazorwasm
$ dotnet watch

Then in Counter.razor change

private void IncrementCount()
{
    count++;
}

to

private void IncrementCount()
{
    try {
      NewMethod();
    } catch (Exception e) {
      Console.WriteLine (e.StackTrace);
   }
}

private void NewMethod()
{
   count++;
   if (count > 3) {
     throw new InvalidOperationException ("count too big");
  }
}

Then click on the button on the Counter page a few times

lambdageek commented 1 year ago

i'm trying to apply my changes to a custom build of the runtime and use it in the Uno app (by overwriting the contents of the Microsoft.NETCore.App.Runtime.Mono.iossimulator-x64 runtime pack), but for some reason the app is crashing with an assertion here:

https://github.com/dotnet/runtime/blob/bf048efe22befbea65ce9a9e2baec3139ec2053b/src/mono/mono/mini/mini-runtime.c#L865

That really shouldn't happen unless we somehow ended up linking multiple copies of Mono (so that there's multiple copies of mono_get_root_domain and the wrong one gets called)

maybe I'm building the runtime wrong (i'm on an arm64 mac, which should in theory be able to make an x64 runtime)

jeromelaban commented 1 year ago

I'm not familiar enough with the procedure to build a custom iOS runtime, but I probably should be more familiar with it in the future :) I'm not sure how I can be of help...

If the PR above is good enough I can validate it on Wasm, if that can help!

lambdageek commented 1 year ago

If the PR above is good enough I can validate it on Wasm, if that can help!

yea that would be useful. although i'm not sure if wasm has the same problems or different problems :D

I'm still trying to understand why in the Uno iossimulator repro something is trying to throw an exception after a hot reload change is applied. The table_info_get_rows crash is masking some other problem - and i'm trying to figure out what that is.

fanyang-mono commented 1 year ago

@jeromelaban with the new zip file, when I following your reproduction steps, I ran into the following error.

/Users/yangfan/Documents/work/issue83860/issue93860/bin/Debug/net8.0-ios/iossimulator-x64/issue93860.app could not be installed on simulator F584177B-85D7-4B39-8627-7EB09B8A69C0.
An error was encountered processing the command (domain=NSPOSIXErrorDomain, code=2):
Simulator device returned an error for the requested operation.
An application bundle was not found at the provided path.
Provide a valid path to the desired application bundle.
Underlying error (domain=NSPOSIXErrorDomain, code=2):
    Failed to install the requested application
    An application bundle was not found at the provided path.

It seems that issue93860.app was not generated. I couldn't find it anywhere. Do you know what I was missing?

jeromelaban commented 1 year ago

@fanyang-mono thanks for trying out the repro! This is curious that you're not getting the app to build. In the terminal that builds the app, are there any errors that are showing up? Otherwise, there's a binlog that is generated in the folder of the mobile project which may contain more information about what's not working.

I'm also wondering, it seems that you're building for x64, are you on an x64 mac as well?

fanyang-mono commented 1 year ago

@jeromelaban Yes, I am using mac x64

jeromelaban commented 1 year ago

@fanyang-mono thanks, then the build error may be specific to your local setup for iOS and iOS workloads, somehow. The binlog I mentioned is very likely to provide additional information.

jeromelaban commented 1 year ago

@lambdageek I just finished validating the fix from https://github.com/dotnet/runtime/pull/94540 on Wasm, it works! Thanks for working on it!

lambdageek commented 1 year ago

@jeromelaban I'm getting some weird behavior when I try to use my local build of Microsoft.NETCore.App.Runtime.Mono.iossimulator-x64 to build the repro app. (What I do is I go into $DOTNET_ROOT/packs/Microsoft.NETCore.App.Runtime.Mono.iossimulator-x64/8.0.0-rc.2.23479.6 and overwrite all the files with the contents of my locally-built nuget. it's crude, but it (usually?) works).

I end up with a crash like this:

14  libmonosgen-2.0.dylib                    0x10f360d12 mono_jit_thread_attach + 178
15  issue93860.Mobile                        0x104d62e65 native_to_managed_trampoline_1(objc_object*, objc_selector*, _MonoMethod**, bool*, unsigned int) + 133 (registrar.mm:15)
16  issue93860.Mobile                        0x104d65a02 -[issue93860_AppHead init] + 50 (registrar.mm:13726)
17  UIKitCore                                0x1340ac44f _UIApplicationMainPreparations + 1585
18  UIKitCore                                0x1340abdf4 UIApplicationMain + 95
19  libxamarin-dotnet-debug.dylib            0x10ed884da xamarin_UIApplicationMain + 58
20  ???                                      0x164305b92 ???
21  libmonosgen-2.0.dylib                    0x129342499 mono_jit_runtime_invoke + 1769
22  libmonosgen-2.0.dylib                    0x1295215e7 mono_runtime_invoke_checked + 135
23  libmonosgen-2.0.dylib                    0x129528625 mono_runtime_exec_main_checked + 117
24  libmonosgen-2.0.dylib                    0x12939f506 mono_jit_exec + 358
25  libxamarin-dotnet-debug.dylib            0x10edccbba xamarin_main + 1898
26  issue93860.Mobile                        0x104d62cc4 main + 52 (main.x86_64.mm:80)

I added some extra printfs and I can tell that I somehow ended up with two copies of mono_get_root_domain. I don't understand how that can be happening (and seemingly only when I drop my own build on top). Does Uno do anything special to the ios build?

Also I still don't understand what is forcing my VS Code to build for iossimulator-x64 rather than iossimulator-arm64. Not sure if it's related. (When I hit ctrl-F5, the build terminal says: Executing task in folder issue83860: dotnet build "/Users/alklig/work/bugs/gh-93860/ver2/issue83860/issue93860.Mobile/issue93860.Mobile.csproj" -f:net8.0-ios -c:Debug -r:iossimulator-x64 -clp:NoSummary -p:GenerateFullPaths=true -p:UnoForceSingleTFM=true -bl:"/Users/alklig/work/bugs/gh-93860/ver2/issue83860/issue93860.Mobile/net8.0-ios-Debug-iossimulator-x64.binlog"). So it's something about the Uno extension, I guess?

lambdageek commented 1 year ago

Turning off the static registrar didn't help. So that's not it.

jeromelaban commented 1 year ago

@lambdageek I'm not sure why x64 is selected, I'm looking into it. But you should be able to build and run directly using dotnet build -f net8.0-ios -r iossimulator-arm64 -t:run (without the debugger though).

jeromelaban commented 1 year ago

Also, you should still be able to test hotreload, if the folder was opened in VS Code.

lambdageek commented 1 year ago

I think the "overwrite libmonosgen-2.0.dylib" trick doesn't quite work for ios. having trouble with it with a plain (arm64) maui app, too. so I'm going to backport the assertion fix to net8 since it is clearly beneficial (our new unit test shows it helps with new code that throws) but I can't guarantee it will actually fix all the problems for Uno's hot reload on ios.

@rolfbjarne is there some way to validate a runtime change with a released version of the ios SDK? everything I'm trying results in suspicious crashes that look like multiple copies of the runtime in the final app.

rolfbjarne commented 1 year ago

@lambdageek I would probably:

  1. Build the iOS app and inspect the binlog to figure out the exact libmonosgen-2.0.a file that's linked into the app.
  2. Replace that libmonosgen-2.0.a file with the one I want to use.

I believe @ivanpovazan had a different (and possibly easier) way to do this though.

ivanpovazan commented 1 year ago

@lambdageek one thing that might help is adding the following target in the project file:

<Target Name="UpdateRuntimePackAndAOTCompiler" 
    AfterTargets="ResolveFrameworkReferences"
    Condition="'$(UseCurrentMainBranch)' == 'true'" >
    <Error Condition="'$(DotnetRuntimeRepo)' == ''" Text="You must specify the location of your dotnet/runtime repository via -p:DotnetRuntimeRepo=&lt;some_repo_dir&gt;"/>
    <ItemGroup>
        <!-- update runtime pack to local build -->
        <ResolvedRuntimePack PackageDirectory="$(DotnetRuntimeRepo)/artifacts/bin/microsoft.netcore.app.runtime.ios-arm64/Release"
            Condition="'%(ResolvedRuntimePack.FrameworkName)' == 'Microsoft.NETCore.App'" />
        <!-- remove AOT compiler reference for ios-arm64 -->
        <MonoAotCrossCompiler Remove="@(MonoAotCrossCompiler)"
            Condition="%(RuntimeIdentifier) == 'ios-arm64'" />
        <!-- update AOT compiler reference to local build -->
        <MonoAotCrossCompiler Include="$(DotnetRuntimeRepo)/artifacts/bin/mono/ios.arm64.Release/cross/ios-arm64/mono-aot-cross">
            <RuntimeIdentifier>ios-arm64</RuntimeIdentifier>
        </MonoAotCrossCompiler>
    </ItemGroup>
</Target>

and then build the app by passing in -p:UseCurrentMainBranch=true -p:DotnetRuntimeRepo=<repo_root> this way you should be able to easily switch between locally built version and the released one.

One thing to note is that the example is configured to target an iOS device.

Hope this helps.

lambdageek commented 1 year ago

Hope this helps.

@ivanpovazan that's very helpful. Unfortunately it only works with an actual device, not with a simulator (I adjusted the obvious RID-specific bits of the .csproj snippet). With the simulator I end up getting errors related to the sdb trampoline - which is usually an indication that some JITing is happening.

@jeromelaban Does Uno hot reload work on an actual device? I tried running the app with Ctrl-F5 and making changes, but I didn't see anything show up. The Uno Platform - Hot Reload and Uno Platform - XAML windows both didn't show any additional output after I changed MainPage.xaml and saved.

lewing commented 1 year ago

@lewing @pavelsavara do we have some way so symbolicate wasm crashes?

The runtime pack should have a dotnet.native.js.symbols to symbolicate @radical can give you more details.

You can also build a Debug runtime locally and update the runtime pack for browser-wasm to use that

radical commented 1 year ago

@lewing @pavelsavara do we have some way so symbolicate wasm crashes?

The runtime pack should have a dotnet.native.js.symbols to symbolicate @radical can give you more details.

We have a symbolicator in src/mono/wasm/symbolicator. You can use it like:

src/mono/wasm/symbolicator$ dotnet run <path-to-dotnet.native.js.symbols-from-the-app> ../data/wasm-symbol-patterns.txt <file-with-trace>

jeromelaban commented 1 year ago

@jeromelaban Does Uno hot reload work on an actual device? I tried running the app with Ctrl-F5 and making changes, but I didn't see anything show up. The Uno Platform - Hot Reload and Uno Platform - XAML windows both didn't show any additional output after I changed MainPage.xaml and saved.

@lambdageek it is supposed to, yes, though because of this issue I wasn't able to test this very far. There may be error messages showing up in the device log, also. You may be able to see more messages when enabling trace logging here:

builder.SetMinimumLevel(LogLevel.Trace);

and here:

builder.AddFilter("Uno.UI.RemoteControl", LogLevel.Trace);
lambdageek commented 10 months ago

@jeromelaban Could you try .NET SDK 8.0.1

jeromelaban commented 10 months ago

@lambdageek I'd like to, but there are no new available workloads available that I could see. VS 17.8.4 still has the 8.0.100 workloads installed. I tried with https://aka.ms/dotnet/maui/main.json, but those are causing install issues on macOS (not sure of the reason).

Would you know of the workload versions I should be using for 8.0.101?

lambdageek commented 10 months ago

@jeromelaban I don't really understand how workloads work. but with a clean install of 8.0.101 on osx-arm64if I run sudo dotnet workload install maui, I see:

...
Installing pack Microsoft.NETCore.App.Runtime.AOT.osx-arm64.Cross.ios-arm64 version 8.0.1...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.AOT.Cross.ios-arm64 version 8.0.1...
...
Installing pack Microsoft.NETCore.App.Runtime.Mono.ios-arm64 version 8.0.1...
Writing workload pack installation record for Microsoft.NETCore.App.Runtime.Mono.ios-arm64 version 8.0.1...
...

which I think means it's getting the servicing versions of the runtime packs

~I have no idea how to discover this info after the workload installation is completed.~

@steveisok tells me the right place to look is sdk-root/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.1/WorkloadManifest.json

lambdageek commented 10 months ago

This is what I see:

% dotnet workload update --print-rollback
==workloadRollbackDefinitionJsonOutputStart==
{
  "microsoft.net.sdk.android": "34.0.52/8.0.100",
  "microsoft.net.sdk.ios": "17.0.8490/8.0.100",
  "microsoft.net.sdk.maccatalyst": "17.0.8490/8.0.100",
  "microsoft.net.sdk.macos": "14.0.8490/8.0.100",
  "microsoft.net.sdk.maui": "8.0.3/8.0.100",
  "microsoft.net.sdk.tvos": "17.0.8490/8.0.100",
  "microsoft.net.workload.mono.toolchain.current": "8.0.1/8.0.100",
  "microsoft.net.workload.emscripten.current": "8.0.1/8.0.100",
  "microsoft.net.workload.emscripten.net6": "8.0.1/8.0.100",
  "microsoft.net.workload.emscripten.net7": "8.0.1/8.0.100",
  "microsoft.net.workload.mono.toolchain.net6": "8.0.1/8.0.100",
  "microsoft.net.workload.mono.toolchain.net7": "8.0.1/8.0.100",
  "microsoft.net.sdk.aspire": "8.0.0-preview.2.23619.3/8.0.100"
}
==workloadRollbackDefinitionJsonOutputEnd==

maybe that's useful?

jeromelaban commented 10 months ago

@lambdageek yes, it's definitely useful, I did not know about --print-rollback, I'll keep it under my belt :D

Still, those were the versions I was using, and I'm having trouble with validating the upgrade from 8.0.100 to 8.0.101 (on macOS, the workload upgrade does not finish properly). Still that's out of the current issue, so I'll probably open a separate issue for this.

I'll test HR for android/iOS and will report back here! Thanks for following up!

jeromelaban commented 10 months ago

So I've tested on android, and the issue is different now:

After a few reloads (like two or three), I get this:

[libc] Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x8 in tid 17526 (Debugger agent), pid 17504 (nyname.UnoApp16)

Interestingly, there's nothing in ADB that is recognizable, but I may not read it properly. So the current issue may be fixed in 8.0.101 for android but I can't test it...

jeromelaban commented 10 months ago

I'm also getting random crashes when starting the app with this:

[nyname.UnoApp16] * Assertion at /__w/1/s/src/mono/mono/metadata/object.c:657, condition `lock->done' not met

But this is unrelated to hot reload.

jeromelaban commented 10 months ago

So the crash issue is https://github.com/dotnet/runtime/issues/96872 (threading related), but if I work around it, I can confirm that the repro steps in https://github.com/dotnet/runtime/issues/93860#issuecomment-1802511533 are not failing anymore, thanks!

lambdageek commented 10 months ago

Ok, the threading thing is because I don't read POSIX manpages closely enough. Closing this issue since the hot reload is working now.