search

Azure / azure-spatial-anchors-samples

Azure Spatial Anchors sample code
Other
293 stars 138 forks source link

iOS app gets into a state where it continuously crashes on app launch since iOS 15.0.x #310

Closed mrwellmann closed 2 years ago

mrwellmann commented 2 years ago

Description We have a verry concerning issue since iOS 15.0.x. Our app is working until a crash with unknown source happens. After this first crash the app crashes every time directly on launch. The only possibility to fix this is to reinstall the app.

This behavior does not happen on every crash of the app it is rather the exception and we have not been able to consistently reproduce it.

The reason I'm writing this here is the crash dump I reference at Additional context which points to eater AzureSpatialAnchors or CoreFoundation.

Steps to reproduce the issue

n/a

Expected behavior

App launches after a crash without issues.

Development information (please complete the following information)

AR Device information (please complete the following information):

Additional context

Find below the crash log. From what I understand, the important parts in this case are Exception Type and Last Exception Backtrace. The exception type is explained here https://developer.apple.com/documentation/xcode/understanding-the-exception-types-in-a-crash-report#EXC_CRASH-(SIGABRT)

Incident Identifier: 11738142-5EE4-4D17-BE34-1295EFEAAFA0
Hardware Model:      iPhone13,3
Process:             FIREDRILL [3310]
Path:                /private/var/containers/Bundle/Application/8C839725-8334-4143-8C1D-206A331CA105/FIREDRILL.app/FIREDRILL
Identifier:          com.firedrillapp.firedrill
Version:             1.8.0 (131)
AppStoreTools:       13A227
AppVariant:          1:iPhone13,3:15
Beta:                YES
Code Type:           ARM-64 (Native)
Role:                Foreground
Parent Process:      launchd [1]
Coalition:           com.firedrillapp.firedrill [1560]

Date/Time:           2021-10-12 13:58:24.7913 +0200
Launch Time:         2021-10-12 13:58:24.4574 +0200
OS Version:          iPhone OS 15.0.1 (19A348)
Release Type:        User
Baseband Version:    2.09.10
Report Version:      104

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Triggered by Thread:  1

Last Exception Backtrace:
0   CoreFoundation                  0x18144805c __exceptionPreprocess + 220 (NSException.m:200)
1   libobjc.A.dylib                 0x199962f54 objc_exception_throw + 60 (objc-exception.mm:565)
2   CoreFoundation                  0x181551538 _CFThrowFormattedException + 116 (CFObject.m:2072)
3   CoreFoundation                  0x181557a20 -[__NSSetM addObject:].cold.1 + 52 (NSSetM.m:165)
4   CoreFoundation                  0x1813d3f6c -[__NSSetM addObject:] + 732 (NSSetM.m:165)
5   AzureSpatialAnchors             0x106ed86ac -[MSAbstractLog addTransmissionTargetToken:] + 208
6   AzureSpatialAnchors             0x106ecad7c -[MSLogDBStorage logsWithCondition:] + 920
7   AzureSpatialAnchors             0x106eca04c -[MSLogDBStorage loadLogsWithGroupId:limit:excludedTargetKeys:completionHandler:] + 876
8   AzureSpatialAnchors             0x106edf6e0 -[MSChannelUnitDefault flushQueue] + 680
9   libdispatch.dylib               0x1810b8c04 _dispatch_call_block_and_release + 32 (init.c:1516)
10  libdispatch.dylib               0x1810ba950 _dispatch_client_callout + 20 (object.m:560)
11  libdispatch.dylib               0x1810c20ac _dispatch_lane_serial_drain + 664 (inline_internal.h:2597)
12  libdispatch.dylib               0x1810c2c10 _dispatch_lane_invoke + 392 (queue.c:3937)
13  libdispatch.dylib               0x1810cd318 _dispatch_workloop_worker_thread + 656 (queue.c:6732)
14  libsystem_pthread.dylib         0x1f18851b0 _pthread_wqthread + 288 (pthread.c:2495)
15  libsystem_pthread.dylib         0x1f1884f50 start_wqthread + 8

Thread 0 name:
Thread 0:
0   UnityFramework                  0x0000000108779b68 _GLOBAL__sub_I_Modules_ParticleSystem_0.cpp + 0 (Modules_ParticleSystem_0.cpp:0)
1   dyld                            0x000000010287c794 ___ZNK5dyld46Loader25findAndRunAllInitializersERNS_12RuntimeStateE_block_invoke + 164 (Loader.cpp:1272)
2   dyld                            0x00000001028b0130 ___ZNK5dyld313MachOAnalyzer18forEachInitializerER11DiagnosticsRKNS0_15VMAddrConverterEU13block_pointerFvjEPKv_block_invoke.309 + 168 (MachOAnalyzer.cpp:3180)
3   dyld                            0x000000010287a490 ___ZNK5dyld39MachOFile14forEachSectionEU13block_pointerFvRKNS0_11SectionInfoEbRbE_block_invoke + 532 (MachOFile.cpp:1310)
4   dyld                            0x0000000102879698 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 168 (MachOFile.cpp:937)
5   dyld                            0x00000001028789f8 dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192 (MachOFile.cpp:1269)
6   dyld                            0x00000001028afcb8 dyld3::MachOAnalyzer::forEachInitializerPointerSection(Diagnostics&, void (unsigned int, unsigned int, unsigned char const*, bool&) block_pointer) const + 148 (MachOAnalyzer.cpp:3084)
7   dyld                            0x0000000102885e68 dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 432 (MachOAnalyzer.cpp:3169)
8   dyld                            0x0000000102882a10 dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 172 (Loader.cpp:1264)
9   dyld                            0x000000010287e3c4 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 208 (Loader.cpp:1304)
10  dyld                            0x0000000102884570 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 124 (Loader.cpp:1314)
11  dyld                            0x000000010287db54 dyld4::APIs::dlopen_from(char const*, int, void*) + 512 (DyldAPIs.cpp:1435)
12  CoreFoundation                  0x00000001814ea09c _CFBundleDlfcnLoadFramework + 140 (CFBundle_Binary.c:610)
13  CoreFoundation                  0x0000000181477058 _CFBundleLoadExecutableAndReturnError + 412 (CFBundle.c:1445)
14  Foundation                      0x0000000182c58978 -[NSBundle loadAndReturnError:] + 428 (NSBundle.m:588)
15  FIREDRILL                       0x000000010272bd54 UnityFrameworkLoad() + 188 (main.mm:10)
16  FIREDRILL                       0x000000010272bdfc main + 36 (main.mm:25)
17  dyld                            0x000000010288da24 start + 520 (dyldMain.cpp:876)

Thread 1 name:
Thread 1 Crashed:
0   libsystem_kernel.dylib          0x00000001b7e989c4 __pthread_kill + 8
1   libsystem_pthread.dylib         0x00000001f188b434 pthread_kill + 268 (pthread.c:1609)
2   libsystem_c.dylib               0x000000018c243f64 abort + 164 (abort.c:118)
3   libc++abi.dylib                 0x0000000199a6abc4 abort_message + 132 (abort_message.cpp:78)
4   libc++abi.dylib                 0x0000000199a5bfd8 demangling_terminate_handler() + 332 (cxa_default_handlers.cpp:71)
5   libobjc.A.dylib                 0x0000000199969064 _objc_terminate() + 144 (objc-exception.mm:701)
6   libc++abi.dylib                 0x0000000199a69f58 std::__terminate(void (*)()) + 20 (cxa_handlers.cpp:59)
7   libc++abi.dylib                 0x0000000199a69ef4 std::terminate() + 64 (cxa_handlers.cpp:88)
8   libdispatch.dylib               0x00000001810ba964 _dispatch_client_callout + 40 (object.m:563)
9   libdispatch.dylib               0x00000001810c20ac _dispatch_lane_serial_drain + 664 (inline_internal.h:2597)
10  libdispatch.dylib               0x00000001810c2c10 _dispatch_lane_invoke + 392 (queue.c:3937)
11  libdispatch.dylib               0x00000001810cd318 _dispatch_workloop_worker_thread + 656 (queue.c:6732)
12  libsystem_pthread.dylib         0x00000001f18851b0 _pthread_wqthread + 288 (pthread.c:2495)
13  libsystem_pthread.dylib         0x00000001f1884f50 start_wqthread + 8

Thread 2:
0   libsystem_pthread.dylib         0x00000001f1884f48 start_wqthread + 0

Thread 1 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x0000000000000000   x2: 0x0000000000000000   x3: 0x0000000000000000
    x4: 0x0000000199a6e0ad   x5: 0x000000016d762440   x6: 0x000000000000006e   x7: 0xffffffff00000300
    x8: 0x0b8316ac07246cb0   x9: 0x0b8316ad6a525cb0  x10: 0x0000000000000002  x11: 0x000000000000000b
   x12: 0x0000000094a29027  x13: 0x0000000014a29000  x14: 0x0000000000000010  x15: 0x0000000000000002
   x16: 0x0000000000000148  x17: 0x000000016d763000  x18: 0x0000000000000000  x19: 0x0000000000000006
   x20: 0x000000000000170b  x21: 0x000000016d7630e0  x22: 0x000000016d7630e0  x23: 0x00000002823c2c40
   x24: 0x00000002818c28e8  x25: 0x0000000000000000  x26: 0x0000000000000000  x27: 0x00000002823c2700
   x28: 0x0000000000000000   fp: 0x000000016d7623b0   lr: 0x00000001f188b434
    sp: 0x000000016d762390   pc: 0x00000001b7e989c4 cpsr: 0x40001000
   esr: 0x56000080  Address size fault

Binary Images:
0x102724000 - 0x10272bfff FIREDRILL arm64  <9fbe0610e25131fa89f47a12938a9888> /private/var/containers/Bundle/Application/8C839725-8334-4143-8C1D-206A331CA105/FIREDRILL.app/FIREDRILL
0x102874000 - 0x1028cbfff dyld arm64e  <d48c31ee061f370ba6f78391a1b53ed8> /usr/lib/dyld
0x1069ac000 - 0x1073a3fff AzureSpatialAnchors arm64  <9a4e30e5448938a6bc5e6010ba9a39fd> /private/var/containers/Bundle/Application/8C839725-8334-4143-8C1D-206A331CA105/FIREDRILL.app/Frameworks/AzureSpatialAnchors.framework/AzureSpatialAnchors
0x10850c000 - 0x10b43bfff UnityFramework arm64  <94831c99cf09374e8ec4b9e378902f6f> /private/var/containers/Bundle/Application/8C839725-8334-4143-8C1D-206A331CA105/FIREDRILL.app/Frameworks/UnityFramework.framework/UnityFramework
0x1810b7000 - 0x1810fcfff libdispatch.dylib arm64e  <959cd6e40ce73022b73c8b36f79f4745> /usr/lib/system/libdispatch.dylib
0x1813af000 - 0x181802fff CoreFoundation arm64e  <6174789ae88c3f5cba39de2e9edc0750> /System/Library/Frameworks/CoreFoundation.framework/CoreFoundation
0x182bd9000 - 0x182eddfff Foundation arm64e  <efbca2ff8b8c3227abbc154ba851d23c> /System/Library/Frameworks/Foundation.framework/Foundation
0x18c224000 - 0x18c2a2fff libsystem_c.dylib arm64e  <059fc305234a3025a1555473cf3a0e9b> /usr/lib/system/libsystem_c.dylib
0x19994d000 - 0x199986fff libobjc.A.dylib arm64e  <6d12ade560653900a6bc80677ccac819> /usr/lib/libobjc.A.dylib
0x199a57000 - 0x199a70fff libc++abi.dylib arm64e  <96151f5b129d3ec79ed0f96c21cf09f6> /usr/lib/libc++abi.dylib
0x1b7e91000 - 0x1b7ec4fff libsystem_kernel.dylib arm64e  <d2476f74d204348d8d386165d0485c7c> /usr/lib/system/libsystem_kernel.dylib
0x1f1884000 - 0x1f188ffff libsystem_pthread.dylib arm64e  <bc1ce0c6a9f2396b9afb623d3acd5881> /usr/lib/system/libsystem_pthread.dylib
mrwellmann commented 2 years ago

I just saw that it is recommend to use AR Foundation 4.0.12 for ASA SDK 2.9 or later https://docs.microsoft.com/en-us/azure/spatial-anchors/how-tos/setup-unity-project?tabs=unity-package-web-ui. Is this still true? We updated to 4.1.7 four moths ago.

mobimar commented 2 years ago

We also have this issue.

The crash reports start showing up on devices even when the app is not running. The crashes stop happening for some time when the app is deleted and reinstalled.

Could it be related to some kind of telemetry that Azure SDK is collecting?

We are using AR Foundation 4.2.0.

mrwellmann commented 2 years ago

Forgot to mention this. I can confirm what @mobimar is saying.

tomkrikorian commented 2 years ago

Can confirm, our app also constantly crashes on iOS 15 and also when not running like @mobimar.

msftradford commented 2 years ago

Thanks so much for reporting this issue @mrwellmann! To answer the above question, ARFoundation 4.1.7 should work just fine as that is the version that we use for testing. Can you (or @mobimar or @tomkrikorian) confirm if this is an issue that repros with our Unity sample app out-of-the-box? Any idea if it happens with our non-Unity iOS sample apps? I'll upgrade my iOS devices now and start looking into it (#36747388).

tomkrikorian commented 2 years ago

I cannot confirm it happens on the Unity Sample from this repo but it does happen outside the sample for us. I can look into it later in the week and try to reproduce with the sample.

mrwellmann commented 2 years ago
MehranAzimi-msft commented 2 years ago

@mrwellmann and @tomkrikorian, thanks again for reporting this issue. We have been trying to repo it on multiple device models running iOS 15 and using ASA's Unity, ObjC and swift samples, but have not been able to repro this yet. As said, it would be great if you can try repro the crash using one of ASA samples.

Our app is working until a crash with unknown source happens. After this first crash the app crashes every time directly on launch. How long does it usually take for your app to get into this crashing state? Is the app crash report and call stack look the same when it crashes on re-launch?

Does your apps use AppCenter frameworks for any purpose, such as crash reporting, telemetry, or distribution?

Based on the symptoms, sounds like the app is leaving some corrupt files on storage after the first crash, which then cause the consecutive re-launches to crash as well. It would be helpful if you check your app's files on your device storage using the Apple's dev tools, as perhaps it can give us a hint about which file is corrupt.

We continue investigating this in our side, and will keep this thread updated.

mrwellmann commented 2 years ago

Yes we use AppCenter Analytics and Crashes for our purpose as well and modified the mainTemplate.gradle for this. See https://github.com/Azure/azure-spatial-anchors-samples/issues/272#issuecomment-882657279

@tomkrikorian @mobimar you do this, too? I can test a build with out it at the start of next week.

mobimar commented 2 years ago

No, we do not use AppCenter Analytics and Crashes directly. I do remember attempting to remove the reference as shown #272, but that resulted in either a build or runtime error, so it remained. Although modifying mainTemplate.gradle is only relevant for Android so I would presume it not to have an effect on this issue.

I think this library could be a very likely source of this failure. AppCenter Analytics crashing while attempting to send telemetry would explain why the program is crashing immediately at the start or when not even started at all. Reinstalling would remove all unsent/cached reports so the first run would work again.

The crashes happen for us when the app is installed from Testflight. As it is in a prerelease state, we can not confirm if it would also happen when installed from the app store. When installing it directly from Xcode, the issue does not seem to be present.

I do not have a device I could create these crashes at the moment. I can look into it next week to try to recreate the failure with the samples project.

mrwellmann commented 2 years ago

Just a quick update crashes keep coming with iOS 15.1.0.

mrwellmann commented 2 years ago

I've create a build with some extra buttons to crash the samples https://github.com/firedrill-gmbh/azure-spatial-anchors-samples/tree/issue-310-iOS-continuously-crashes but I was not able to to reproduce the bug during my about 20 crashes in different situations.

If AppCenter is the reason it will be really hard to reproduce it because I don't know about the implementation of AppCenter in ASA SDK and I don't know when it will send a report and when not I also can't deactivate it or postpone it.

I've forwarded this to AppCenter support and I will read a bit more on the AppCenter documentation when I have time to continue on this.

@mobimar The crashes happen for us when the app is installed from Testflight. As it is in a prerelease state, we can not confirm if it would also happen when installed from the app store. When installing it directly from Xcode, the issue does not seem to be present.

I am certain it is not a Testflight specific issue. We have the same issue when sharing a build with Unity Cloud Build. One thing wich is noted on the App Center documentation is that "The SDK won't forward any crash logs if you've attached the debugger. Make sure the debugger isn't attached when you crash the app."

tinanigro commented 2 years ago

Hello Azure team,

Any update on this issue? Even thought it might end up being unrelated to TestFlight, I believe it may useful that you try to upload your sample on TestFlight if you haven't already.

For some reason I don't understand, it seems that we encounter many more occurrences of these crashes when the app is installed through TestFlight rather than locally built and deployed.

Furthermore, I want to emphasise how exceptionally weird this bug is. More often than not, right after restarting our iPhone devices, an iOS dialog is displayed saying our app crashed. It wasn't even launched or any similar user action, and our app obviously doesn't have any permission that would let it run in the background. This is extremely surprising to me as I never had such a weird behaviour on iOS, both as an iOS developer and an iOS user.

As a result the simple fact of installing the app and running it at least once, will result in several iOS crash dialogs being displayed at any time (after restarting the device, or at random times during day to day use). Sometimes that same dialog will be displayed up to 5 (if not more) times in a span of just 2 or 3 seconds, and therefore are displayed on top of each other.

As you can see, for many reasons laid above by other users and I, this issue is particularly concerning and makes it impossible to distribute our app at the moment; this is why we are willing to help in any way we can in other to help you diagnose and investigate the matter.

mrwellmann commented 2 years ago
  1. The issue also happens with App Store installations. The app will immediately crash on launch but the good thing is it will not trigger any iOS crash dialogs.
  2. I've opened a ticket at Unity AppCenter Github https://github.com/microsoft/appcenter-sdk-unity/issues/503 and at the AppCenter Support directly (Analytics Issue #KVAT3A1K) they both told me they need verbose logs of the crash to further help with it.
    1. I'm unsure if anything I'm doing on our AppCenter implementation will have an effect on the ASA AppCenter implementation
    2. Anyway I will try to get those logs by activating the verbose logs in our implementation of AppCenter. But it will probably be not his week. If anyone else wants to go ahead, to be able to use AppCenter your self you need to modify your mainTemplate.gradle https://github.com/Azure/azure-spatial-anchors-samples/issues/272
MehranAzimi-msft commented 2 years ago

Thanks for your posts @mrwellmann and @tinanigro. FYI, we have not been able to repro this issue on our side yet. However, we are working on removing the ASA SDK dependency on AppCenter for SDK telemetry and replace it with 1DS library. We are also adding new APIs that allows an app to disable telemetry. These changes will be in next SDK 2.11 release. As soon as we have an ETA for this release, we will share it here.

tomkrikorian commented 2 years ago

Hi, Any rough estimate on the release ? Or can we modify the current SDK to disable the telemetry somewhere ? I'm sure you're doing your best but it's been a month we can't release our app because of these crashes šŸ™

MehranAzimi-msft commented 2 years ago

@tomkrikorian we are treating this issue with high priority and actively working on it. We roughly estimate to have the fix released by end of next week 11/19/21.

tomkrikorian commented 2 years ago

Amazing, thank you šŸ™

tinanigro commented 2 years ago

That's terrific news! Thank you for the fast answer and the transparency with regards to the ETA. This is very appreciated and definitely helps building trust between the ASA team and developers! šŸ˜

MehranAzimi-msft commented 2 years ago

ASA SDK 2.11.0 was just released. See release notes posted here. SDK dependency on AppCenter for telemetry is replaced with 1DS on iOS platform in this release, and so we expect the issue reported here is now fixed in SDK 2.11.0. Also note there is a new API that allows apps to disable telemetry. If app code calls CloudSpatialAnchorSession::EnableTelemetry(false) before calling CloudSpatialAnchorSession::Start(), then no telemetry is logged by ASA SDK.

Please let us know if this fixes the crash in your apps.

tomkrikorian commented 2 years ago

Hi, It does fix the crash that happens even if the app is not launched. I still have a crash that happens often but will open a new ticket for it.

update : new ticket opened here : https://github.com/Azure/azure-spatial-anchors-samples/issues/323

phamqduc commented 2 years ago

Thank you for letting us know that the release addresses the previous crash, @tomkrikorian . We will look into your new ticket.