Hangs and Unresponsive Behavior in App Crashes (Ditto SDK and Rust 1.65+ on Apple arm64 Devices)

[konstantinbe]

Description

We are experiencing hangs when hitting a crash in an iOS app, if used together with our Ditto SDK. The core of our SDK is built with Rust. The problem surfaces when we build our Core with Rust starting with version >= 1.65 and only on arm64 devices (Mac & iOS).

To demonstrate the issue, we’ve prepared a minimal Mac app: CrashReporterFreeze. Complete the following steps to reproduce the unresponsive behavior:

Repro Steps

Trigger a crash within an app by doing the following steps:

1. If not already installed, install Git LFS.
2. Clone CrashReporterFreeze on either a “M1” or “M2” Mac.
3. In Xcode, open CrashReporterFreeze.xcodeproj, select target CrashReporterFreeze, build & run the app. Once running, the app appears in the Dock as an icon.
4. Click and drag the app icon to the permanent section in the Dock.
5. In Xcode, stop the app from running.
6. Open the Finder > then go to Applications > Utilities > open the Console app > and then click Start Streaming.
7. Type "CrashReporterFreeze" into the search field, and then select the Process query item that appears.
8. From the Dock, click to launch the just built CrashReporterFreeze app.
9. Once launched, click Crash!. At this time, you may notice the app becoming unresponsive, as indicated by the “beachball of death” and the debug logs being printed within the Console.app. These logs were embedded by us deep within the crash processing functionality of the PLCrashReporter.

Although most of the time the crash processing eventually completes, sometimes it seems to hang indefinitely.

If you want to test the exact same version of the Ditto SDK built with different versions of Rust, repeat the previous steps on any of the following commits in the CrashReporterFreeze repo:

79d8fa2 Add DittoObjC v3.0.9 built with Rust 1.70 | Bad Konstantin Bender (4 weeks ago) (HEAD -> main, origin/main)
ec72757 Add DittoObjC v3.0.9 built with Rust 1.65 | Bad Konstantin Bender (4 weeks ago)
4ede7dd Add DittoObjC v3.0.9 built with Rust 1.64 | Good Konstantin Bender (4 weeks ago)

Debugging

While there may be alternative approaches to debugging the issue, the following approach is the method that has proven most effective for us.

In this approach, we combine the use of the Console.app and manually attach the Xcode debugger following the crash by doing the following:

1. In Xcode, build the app by pressing Command + B.
2. From the Dock, launch the CrashReporterFreeze app and then click Crash!.
3. Monitor the Console app to observe the unresponsive behavior: The crash processing repeatedly prints the same debug message. 
4. In Xcode:
   1. Attach the debugger by clicking Debug > Attach to Process > CrashReporterFreeze.
   2. Use search to locate the line for "[DEBUG] entry_idx," and then set a breakpoint on it.
   3. Eventually, Xcode should hit that breakpoint.

Details

1. Which SDK version are you using? Latest commit on master of the PLCrashReporter repo.
2. Which OS version did you experience the issue on? macOS 13.
3. Which CocoaPods/Carthage/Xcode version are you using? Xcode 14.3.
4. What device version did you see this error on? Were you using an emulator or a physical device? Apple MacBook Pro 16" M1 Pro, 32 GB RAM, 2 TB SSD.
5. What language are you using?
   • [ ] Objective C
   • [ ] Swift
6. What third party libraries are you using? Ditto SDK.

[MikhailSuendukov]

Hi @konstantinbe and thank you very much for the detailed information. Unfortunately, I was unable to open your project with XCode 14.3.1. It is possible that the file is broken. Could you please fix it or maybe rebuild the project, so I can test it on my side?

[konstantinbe]

@MikhailSuendukov thank you for trying it out. There are two things at play here:

1. After installing git lfs, you have to call git lfs install separately. Here is what brew says:

If I leave this part out, I get exactly the error as you previously posted.

2. Everything in the CrashReporterFreeze is apparently git-lfs-ed. The migration script didn't work as I expected it to. Only the large binary files (DittoObjC) should be lfs-ed.

I've fixed the repo, please try again and make sure to run git lfs install first. Please let me know if you still have problems and I'll be happy to assist further.

[MikhailSuendukov]

It's very strange, but I still get this error. I use these commands to clone your repository:

git lfs install
git clone https://github.com/getditto/CrashReporterFreeze.git

After that I try to open the project file and the same error appears.

[konstantinbe]

Strange, sorry for the trouble. I've uploaded a non-LFS version of the repo to S3:

https://konstantinbe.s3-eu-central-1.amazonaws.com/Shared/CrashReporterFreezeNoLFS.zip

Could you try that?

[AnatolyPristensky]

Hello @konstantinbe, I was able to reproduce your issue. Investigating, will update this thread once get results.

[okdistribute]

Thank you for taking a look!

[AnatolyPristensky]

@konstantinbe I've spent a lot of time on digging into this issue and it's really hard to identify a root cause. It mainly happens due to integration of DittoObjC library which is built with a particular Rust version. Probably you should ask about your case in the Rust repository as well. For now, this issue has low priority and I'm closing it as won't fix.

Also, we always welcome any contributions.

[konstantinbe]

@AnatolyPristensky thank you for taking the time to dig into this. 💚 We'll try to raise this with the Rust folks and take it from there.