Open TimBurik opened 1 week ago
Could this issue be related to https://github.com/Unity-Technologies/mono/pull/1796?
Additionally, some of the repots include GWP-ASan stacktrace:
backtrace:
#00 pc 0x00000000000d770c /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_method_to_ir+5714)
dotnet/android#1 pc 0x00000000000ca3ac /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (inline_method+4825)
dotnet/android#2 pc 0x00000000000e3e74 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_method_to_ir+7873)
dotnet/android#3 pc 0x00000000000bb2a8 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mini_method_compile+3498)
dotnet/android#4 pc 0x00000000000bd8e4 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_jit_compile_method_inner+4132)
dotnet/android#5 pc 0x00000000000c25b0 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (jit_compile_method_with_opt+2813)
dotnet/android#6 pc 0x00000000000c1a1c /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_jit_compile_method+2903)
dotnet/android#7 pc 0x0000000000152194 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (common_call_trampoline+628)
dotnet/android#8 pc 0x0000000000151cf8 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_magic_trampoline+769)
dotnet/android#9 pc 0x0000000000004300
Cause: [GWP-ASan]: Buffer Overflow, 11 bytes right of a 16-byte allocation at 0x7ad99d5ff0
allocated by thread 14095:
#00 pc 0x0000000000054a44 /apex/com.android.runtime/lib64/bionic/libc.so (gwp_asan::AllocationMetadata::CallSiteInfo::RecordBacktrace(unsigned long (*)(unsigned long*, unsigned long))+84)
dotnet/android#1 pc 0x00000000000550b8 /apex/com.android.runtime/lib64/bionic/libc.so (gwp_asan::GuardedPoolAllocator::allocate(unsigned long, unsigned long)+600)
dotnet/android#2 pc 0x0000000000045ae4 /apex/com.android.runtime/lib64/bionic/libc.so ((anonymous namespace)::gwp_asan_calloc(unsigned long, unsigned long)+192)
dotnet/android#3 pc 0x0000000000046450 /apex/com.android.runtime/lib64/bionic/libc.so (calloc+124)
dotnet/android#4 pc 0x00000000001d6d94 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (monoeg_malloc0+131)
dotnet/android#5 pc 0x0000000000253dc4 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_metadata_type_dup_with_cmods+6195)
dotnet/android#6 pc 0x0000000000201f00 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (inflate_generic_type+724)
dotnet/android#7 pc 0x0000000000201ccc /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (inflate_generic_type+864)
dotnet/android#8 pc 0x00000000002020cc /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_class_inflate_generic_type_checked+979)
dotnet/android#9 pc 0x000000000023afb8 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (inflate_generic_signature_checked+626)
dotnet/android#10 pc 0x000000000023c994 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_method_signature_checked_slow+1804)
dotnet/android#11 pc 0x000000000023cc50 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_method_signature_internal_slow+775)
dotnet/android#12 pc 0x00000000000df750 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_method_to_ir+786)
dotnet/android#13 pc 0x00000000000bb2a8 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mini_method_compile+3498)
dotnet/android#14 pc 0x00000000000bd8e4 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_jit_compile_method_inner+4132)
dotnet/android#15 pc 0x00000000000c25b0 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (jit_compile_method_with_opt+2813)
dotnet/android#16 pc 0x00000000000c1a1c /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_jit_compile_method+2903)
dotnet/android#17 pc 0x0000000000152194 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (common_call_trampoline+628)
dotnet/android#18 pc 0x0000000000151cf8 /data/app/~~sLg-sWOLdAwq7fv0Z5P4iQ==/<app.bundle.id>-Oub7eEv0VVrZY4VBnnyfgg==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_magic_trampoline+769)
dotnet/android#19 pc 0x0000000000004300
dotnet/android#20 pc 0x0000000000008c54
dotnet/android#21 pc 0x00000000000052a0
dotnet/android#22 pc 0x00000000000051e0
dotnet/android#23 pc 0x0000000000004b54
dotnet/android#24 pc 0x0000000000007de8
dotnet/android#25 pc 0x0000000000007ba0
dotnet/android#26 pc 0x00000000000073bc
dotnet/android#27 pc 0x00000000000070f8
dotnet/android#28 pc 0x0000000000005590
dotnet/android#29 pc 0x00000000000051b4
dotnet/android#30 pc 0x00000000000067b8
dotnet/android#31 pc 0x0000000000006508
dotnet/android#32 pc 0x0000000000000294
dotnet/android#33 pc 0x00000000003dc3e0 /data/misc/apexdata/com.android.art/dalvik-cache/arm64/boot.oat (art_jni_trampoline+112)
@vitek-karas @akoeplinger assigning to you since I have no idea of the current area "owners" in MonoVM :) Please redirect as needed (and possibly move to dotnet/runtime
, I have no permission to do so) Thanks :)
Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger See info in area-owners.md if you want to be subscribed.
Tagging subscribers to this area: @lambdageek, @steveisok See info in area-owners.md if you want to be subscribed.
@steveisok could you please take a look at the crashlogs? We have a pretty big crash group for this issue in production, and at the moment it doesn't look like we can do anything on our side. Or maybe we could - please let us know.
I tried to get some line numbers for this stack trace:
#00 pc 0x00000000000d770c /data/app/~~-rAQpqUqyTqj-baEpS-Wbw==/<app.bundle.id>-IoSfRyrENCUl2s9Fv_CpjA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_method_to_ir+5714)
and lldb says
(lldb) image lookup --address 0xd770c
Address: libmonosgen-2.0.so[0x00000000000d770c] (libmonosgen-2.0.so.PT_LOAD[0]..text + 131312)
Summary: libmonosgen-2.0.so`mono_method_to_ir + 21700 [inlined] try_prepare_objaddr_callvirt_optimization + 12 at method-to-ir.c:5709:6
libmonosgen-2.0.so`mono_method_to_ir + 21688 at method-to-ir.c:7132:35
which is this line
inlined from here
which looks sus because param_types [n]->data.klass
doesn't check that param_types[n]->type
to make sure that that the data.klass
union field is valid - so it's passing a bogus "klass" pointer and then trying to call mono_class_is_def
on it.
@steveisok next steps here would be to make some kind of test case (I guess try to get an ldarg ; callvirt
instruction sequence where the argument type is something not class-like - probably an array. Update not an array. needs something more obscure...
Update2 ok, !klass || !mono_class_is_def(klass)
is actually very annoying to crash... klass
has to be non-NULL but pointing at something that can't be nicely derefed since mono_class_is_def
is just klass->class_kind == MONO_CLASS_DEF
... but all the MonoType:data
fields are pointers so often they're either null or they point to something with at least ~4 fields - so offsetof(klass,class_kind)
is pointing to valid memory even if the value there is garbage.
So I'm not actually sure how to intentionally crash us here.
Ok, after a couple hundred iterations, I got this to crash inside try_prepare_objaddr_callvirt_optimization
with a bad klass
:
using System;
using System.Collections;
using System.Runtime.CompilerServices;
public class C {
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public string M(int y, int p, int q, int n, int m, int o, int[,] s) {
return (s as IEnumerable).GetEnumerator().ToString();
}
[MethodImpl(MethodImplOptions.NoInlining)]
public string Q(int[,] s) {
return M(0, 0, 0, 0, 0, 0, s);
}
public static void Main()
{
C c = new();
int[,] x = new int[5,10];
Console.WriteLine(c.Q(x));
}
}
So there's definitely bad inputs here. I'm not 100% confident that this is the exact same crash as OP is reporting because while the arm64 stack traces makes sense, my lldb is giving me very weird debug info for the arm32 crashes. I suspect I'm not configuring it correctly, but maybe the actual issue is that i'm misinterpreting the arm64 crashes too and I just happened to have found a different issue.
But fixing the bug that I found I think the right move is to change try_prepare_objaddr_callvirt_optimization
to just take a MonoType*
argument ¶m_types[n]
and then call mono_class_from_mono_type_internal
on it to get a MonoClass
.
@steveisok
we also hit this crash, our backtrace:
backtrace:
#00 pc 000fb2d9 /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (mono_method_to_ir [src/mono/mono/mini/method-to-ir.c : 7137 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#01 pc 000ed1dd /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (inline_method [src/mono/mono/mini/method-to-ir.c : 4825 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#02 pc 00110e89 /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (mono_method_to_ir [src/mono/mono/mini/method-to-ir.c : 9369 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#03 pc 000ed1dd /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (inline_method [src/mono/mono/mini/method-to-ir.c : 4825 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#04 pc 00107766 /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (mono_method_to_ir [src/mono/mono/mini/method-to-ir.c : 7873 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#05 pc 000ddd12 /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (mini_method_compile [src/mono/mono/mini/mini.c : 3498 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#06 pc 000e053a /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (mono_jit_compile_method_inner [src/mono/mono/mini/mini.c : 4132 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#07 pc 000e5306 /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (jit_compile_method_with_opt [src/mono/mono/mini/mini-runtime.c : 2884 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#08 pc 000e4668 /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (mono_jit_compile_method [src/mono/mono/mini/mini-runtime.c : 2903 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#09 pc 0018115c /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (common_call_trampoline [src/mono/mono/mini/mini-trampolines.c : 628 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#10 pc 00180d5e /data/app/~~rLVoPlpXa04eWTkHl9dGmw==-owS4rz2lnMAezWGjuCFh_w==/lib/x86_64/libmonosgen-2.0.so (mono_magic_trampoline [src/mono/mono/mini/mini-trampolines.c : 769 + 0x0]) (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
#11 pc 00000395 <anonymous:40d53000> (BuildId: 0c48500975dc10eb3d45f354c84338c43cc0acfc))
Is there any way to work around this? what method signature can lead this crash?
It has to be a method that calls an interface or virtual method on one of its own arguments. And the type of the argument has to be something other than a class. I could only get it to happen with a multi-dimensional array. The method IL has to have ldarg+callvirt
Android framework version
net8.0-android, net9.0-android
Affected platform version
.NET 8.0.303, .NET 9-rc2
Description
After switching from Xamarin.Android to .Net8 (and later to .Net9) a new native crash group has appeared in the GooglePlay Console, which seems to be related to the Jit compiler optimizations.
Here are some stacktrace examples from .Net8 releases:
and from .Net9 release:
Steps to Reproduce
Unfortunately, we don't have exact steps to reproduce. We can't (at least reliably) reproduce it during testing phase, but we see a crash group in the GooglePlay Console with a noticeable counter every time we try
.net-android
instead ofXamarin.Android
.Did you find any workaround?
No workaround found yet
Relevant log output
No response