iOS 14 + Xcode 12 (Beta 3, Beta 4, Beta 5 & Beta 6): When Realm is stored in a shared app group container, backgrounding the app triggers: Message from debugger: Terminated due to signal 9

[kunalsood]

Goals

Have my app continue to run normally, when backgrounded, with Realm stored in a shared app group container.

Expected Results

Same as "Goals" [above].

Actual Results

When, I run the app on an actual device, and swipe up while my app is in the foreground to send it to the background, the app quits with the following message in Xcode:-

Message from debugger: Terminated due to signal 9

Additionally: This only happens if I'm holding a reference to a live RLMResults or a notification token (those are the two things I have tested this with so far).

Interestingly, when I run the project with the default realm configurations (i.e. Realm is stored in the documents directory), this issue disappears.

It might be worth mentioning: It appears that the crash report produced on device for this issue when debugger isn't attached shows: Termination Reason: Namespace RUNNINGBOARD, Code 0xdead10cc, which would indicate that the application is being terminated by the OS because it held on to a file-lock/database-lock during suspension. (Could be un-related, though there is nothing other than Realm in my project that would hold on to a file-lock/database-lock ever).

Steps for others to Reproduce

Build and run the sample project using Xcode 12 (Beta 3, Beta 4, Beta 5, or Beta 6) on an actual device running iOS 14 (Beta 3, Beta 4, Beta 5, or Beta 6). (I have tested this on an iPhone & an iPad)

Code Sample

Here is a (quick & dirty) sample project: https://github.com/kunalsood/KSRealmTerminationSample

Most of the relevant code in the project is in: ViewController.m file & one realm model object file KSSampleObject.h.

Version of Realm and Tooling

Realm framework version: 5.3.2 (pre-built dynamic framework) UPDATE: Sample project has now been updated to include Realm using SPM.

Realm Object Server version: N/A

Xcode version: 12.0 (Beta 3, Beta 4, Beta 5 & Beta 6)

iOS/OSX version: iOS 14 (Beta 3, Beta 4, Beta 5 & Beta 6)

Dependency manager + version: N/A

[meteochu]

+1 on having identical issues. Group Container, Xcode 12b3 + iOS14b3, and crashing when backgrounded.

I'm using the latest RealmSwift (5.3.2.) with Swift PM.

[oncezw]

+1 same issues in + iOS14b3, and crashing when backgrounded.

[bguidolim]

There is a bug report filed for this issue: https://developer.apple.com/forums/thread/655225 I don't think it's a Realm bug.

I'm facing the same problem, it's about locked files in the shared folder.

[SquaredTiki]

+1, author of that thread on the Apple Dev Forums.

Would recommend folks dupe the existing feedback reports (FB8128103 & FB8116961) if they can. May well bring it to Apple's attention more quickly. Especially if you have a sample project repro'ing as my report didn't include one.

Title iOS 14 Beta 3 triggers consistent 0xdead10cc terminations impacting binary compatibility

Component UIKit

Type Application Crash

Description DESCRIPTION OF PROBLEM We have observed that as of beta 3 our app now consistently & immediately crashes as soon as it is suspended (i.e. the user 'closes' the app). The crash log has an exception reason of 0xdead10cc indicating that the crash is due to the app holding on to a file lock or sqlite database lock during suspension.

This crash occurs on all recent versions of our app but only on beta 3 and does not occur at all before beta 3. We can consistently reproduce the issue on iOS 14 beta 3 (18A5332f) but are unable to do so on previous betas of iOS 14 or any version of iOS 13.

The console log narrows down the file triggering the crash on suspension:

[application<…>:3879] Terminating with context: <RBSTerminateContext| domain:15 code:0xDEAD10CC explanation:[application<…>:3879] was suspended with locked system files: /var/mobile/Containers/Shared/AppGroup/A66EB78A-2BBC-49D4-BDEA-6A2AF7E8A5A6/default.realm.lock not in allowed directories: /var/mobile/Containers/Data/Application/E1435A44-ABC6-4254-B547-B5423D9FCAB1 /var/mobile/Containers/Data/Application/E1435A44-ABC6-4254-B547-B5423D9FCAB1/tmp reportType:CrashLog maxTerminationResistance:Interactive>

This points to Realm's default.realm.lock being the locked file triggering the crash, suggesting this file is not permitted as it sits within the App Group container as opposed to the app's own container (which I presume is the first 'allowed directory').

Again, whilst this explains the cause of the crash on beta 3 it doesn't explain why the crash has only just begun occurring on this build of iOS 14 (beta 3, 18A5332f). This suggests to me that there may have been a system level change to cause this and potentially break binary compatibility with existing versions of apps (including others than our own that use Realm within an App Group).

STEPS TO REPRODUCE

• Launch an app that uses Realm with App Groups (e.g. <"Your App on the App Store" or "Sample Project Attached">)
• Suspend the application
• Runningboard terminates the app
• Console log indicates a 0xdead10cc termination reason due to default.realm.lock still being locked

PLATFORM AND VERSION iOS 14.0 Beta 3 (18A5332f) Devices: All

NOTES Also raised on the developer forums and discussed with Quinn “The Eskimo!” here: https://developer.apple.com/forums/thread/655225.

[kunalsood]

@SquaredTiki Thanks for including details of for Feedback to Apple. I'm in the process of submitting a dupe with a sample project. Will post the feedback number on your Apple Dev Forums thread when done.

[n1kitus]

+1 identical issues in iOS 14b3. The description of the problem perfectly fits our case.

[zacwest]

This crash continues in 14.0 b4.

[indirect]

I’ve heard this kind of crash can often be fixed with task assertions, are those assertions getting used here?

https://developer.apple.com/documentation/uikit/uiapplication/1623031-beginbackgroundtask

[gongzhang]

I believe this is an iOS bug, since:

1. only reproduce on physical device running iOS 14b3/b4
2. when crash with a debugger attached, Xcode just ends the debug session silently without showing any error messages. (except the termination signal message)

[gongzhang]

I’ve heard this kind of crash can often be fixed with task assertions, are those assertions getting used here?

https://developer.apple.com/documentation/uikit/uiapplication/1623031-beginbackgroundtask

I am also trying to walk around this issue. But it's difficult when you have to save the db in app group container. 😟

[kunalsood]

I’ve heard this kind of crash can often be fixed with task assertions

@indirect That would be the case if we were crashing while in an active a write transaction. We are not. The app crashes even if I'm just holding on to an instance of a live RLMResults, RLMArray, RLMObject, RLMRealm, or RLMNotificationToken. I do not think it makes sense to wrap any of these in task assertions.

What you have heard is probably applicable for versions of iOS that came before iOS 14b3. Prior to iOS 14b3, this issue used to occur in cases that can broadly be divided in two:-

1. If you were doing a long write transaction when the app entered background, and have either not requested a task assertion, or the additional time allotted by the system to your app ended before you ended your write transaction.
2. You were doing a write transaction from an extension without using performExpiringActivityWithReason:usingBlock: method of NSProcessInfo, and the extension's process was killed by the system, or, performExpiringActivityWithReason:usingBlock: failed to acquire a task assertion (it fails more often than not in my experience), or, the additional time allotted by the system to your extension's process ended before you ended your write transaction.

are those assertions getting used here?

I use them where they makes sense.

This is either a bug introduced in iOS 14b3, or (Apple has decided that) this is the new expected behavior. I'm inclined to believe that it's the former, because as has already been said, this only started happening in iOS14b3, and when this happens with the debugger attached, Xcode just ends the debug session silently without showing any error messages. The latter would make use of Realm unfeasible for apps with extensions that need access to the database.

[meteochu]

Just checking in, has anyone gotten updates from Apple about their submitted feedback and if we know this is a new system behaviour or just a system regression?

[kunalsood]

Just checking in, has anyone gotten updates from Apple about their submitted feedback and if we know this is a new system behaviour or just a system regression?

That's a negative from me on both. Nothing yet for my feedback, still marked as Open.

[gongzhang]

Does anyone have the idea about how to reproduce the bug without Realm? Maybe like retain a file lock in shared folder?

[owenzhao]

Does anyone have the idea about how to reproduce the bug without Realm? Maybe like retain a file lock in shared folder?

The question is not why Ream puts a lock file there. The question is why Apple doesn't allow a lock file in App Group folder. Is there any harm?

[meteochu]

Does anyone have the idea about how to reproduce the bug without Realm? Maybe like retain a file lock in shared folder?

The question is not why Ream puts a lock file there. The question is why Apple doesn't allow a lock file in App Group folder. Is there any harm?

It is possible that this is the new expected behaviour to prevent multiple apps for writing to the same file by checking for file locks (since App group containers can be accessed by more than one process at a time). However, I'd like to believe that it's a regression since it is not being blocked by linked-on checks for iOS 14 and is instead affecting current apps.

[zhangcx627]

+1 same issues in + iOS14b3, and crashing when backgrounded.

[SquaredTiki]

No updates on my bug report to Apple but I can see it says that there are less than 10 'recent' dupes, would recommend folks file a duplicate bug report if they haven't already to help bring increased attention (example/template here).

[gongzhang]

Does anyone know which exact line of code in Realm trigger this bug?

I think it's better to reproduce the bug in a clean and simple demo project without importing the entire Realm framework. I believe it will help Apple to identify the bug.

[zacwest]

This crash continues in 14.0 b5. Probably time to consider migrating off Realm for iOS 14.

[owenzhao]

This crash continues in 14.0 b5. Probably time to consider migrating off Realm for iOS 14.

There are other three ways:

1. Realm adds a new config to put lock file in the app's container, separately from the database file.
2. Realm releases the lock file every time the Realm.write state is finished. I had a test by putting a file named "test.lock" in App Group's container using FileMananger. The app didn't crash as Realm did.
3. Using iCloud syncs the realm database between app and extensions, avoid of using App Group with Realm.

[tgoyne]

The lock file is placed in the app container.

This appears to be Apple's "fix" for the problem where switching between apps sharing a Realm file in a container could deadlock if done during a write transaction because the now-backgrounded app would hold the write transaction forever. Simply killing the app when that happens isn't exactly an ideal solution, but it does ensure things keep working. The problem appears to be that it's being overly aggressive and is also killing the app if a shared lock is held. There's nothing inherently wrong with a shared lock continuing to be held by a suspended process.

[kunalsood]

@tgoyne Say if Apple has decided that this is the new expected behaviour. Is there anything you (or I) can do to work around this?

[tgoyne]

There probably isn't anything easy. I suspect Apple's intended answer is "just close the file when you get the transition to background notification", which is very difficult when multiple threads are involved. The big important thing that the shared lock is doing is letting us clean up after crashes rather than forever holding onto data which was only needed by a process that is no longer running, so we could try to come up with some other approach to doing that.

[owenzhao]

Here is my work around. My app won't crash any longer. If you use this, take your own risk.

func applicationWillResignActive(_ application: UIApplication) {
    // Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone call or SMS message) or when the user quits the application and it begins the transition to the background state.
    // Use this method to pause ongoing tasks, disable timers, and invalidate graphics rendering callbacks. Games should use this method to pause the game.
    let fm = FileManager.default
    let url = lockfile()

    if fm.fileExists(atPath: url.path) {
        try? fm.removeItem(at: url)
    }
}

func applicationDidBecomeActive(_ application: UIApplication) {
    // Restart any tasks that were paused (or not yet started) while the application was inactive. If the application was previously in the background, optionally refresh the user interface.

    if !FileManager.default.fileExists(atPath: lockfile().path){
        let _ = try! Realm(configuration: Realm.sharedConfiguration)
    }
}

private func lockfile() -> URL {
    let fm = FileManager.default
    let url = fm.containerURL(forSecurityApplicationGroupIdentifier: MyDefaults.sharedSuitName)
    let lockfileName = "default.realm.lock"
    let lockfileURL = URL(fileURLWithPath: lockfileName, isDirectory: false, relativeTo: url)

    return lockfileURL
}

Just remove the lock file when app no longer active and recreate the lock file when it become active.

[kunalsood]

@tgoyne Thanks for your response.

@owenzhao The lock file is there for a good reason, simply deleting it will probably cause more problems (probably even data corruption?) than it will solve.

[owenzhao]

@tgoyne Thanks for your response.

@owenzhao The lock file is there for a good reason, simply deleting it will probably cause more problems (probably even data corruption?) than it will solve.

I know my app better than you. For my app, this is a proper way. It is app's developer to decide whether to use this work around.

The better work around is to use iCloud syncing between app and extensions. But that will take much more work.

[gongzhang]

Hey guys, I just reproduce the bug without Realm. It's all about holding flock in App Group directory. You can follow these steps to quickly reproduce the weird crash:

1. Create a new iOS App project (Objective-C) using Xcode.
2. Add an App Group container in project setting page.
3. Paste this code snippet in application:didFinishLaunchingWithOptions::

    // 1. prepare a non-empty file under App Group directory
    NSFileManager* fileManager = [NSFileManager defaultManager];
    NSURL* dir = [fileManager containerURLForSecurityApplicationGroupIdentifier:@"group.xxx.xxx....."]; <= your group id here
    NSURL* fileUrl = [dir URLByAppendingPathComponent:@"file"];
    [fileManager createFileAtPath:[fileUrl path]
                         contents:[@"some data..." dataUsingEncoding:(NSUTF8StringEncoding)]
                       attributes:nil];

    // 2. hold a file lock
    int fd = open([fileUrl path].UTF8String, O_RDWR);
    int ret = flock(fd, LOCK_SH);

4. Debug the project on a physical device running iOS 14 b3/b4/b5.
5. The app will be killed after you return to home screen.
6. If you unlock the file by calling flock(fd, LOCK_UN) before the app enters suspended state, the app won't be killed by iOS.

---

Note that:

• This only crash on a physical device, not a simulator.
• Xcode does not handle it like a normal crash. It just print a termination message in console and ends the debug session gracefully.

[finnschiermer]

With respect to removing the lock file in a running system: Don't

The lock file is what allows us to coordinate access to the realm file itself among multiple Core DB objects (realm-core v6 and onwards) or multiple SharedGroup objects (pre realm-core v6). Removing the lock file is guaranteed NOT to work, as soon as multiple such objects need to coordinate access.

Realm does not guarantee any particular coupling between processes/threads and underlying use of DB/SharedGroup objects. We may also want to change any such coupling in the future, just as we've changed it recently. If you employ a scheme which removes or tinkers with the lock file while the app is running, your code may break with next version of Realm. Or it may break with the current version - only you haven't noticed yet :-O

Point is: You do not have control over when multiple DB objects are in play. Sorry.

Don't mess with the lock file.

[gongzhang]

With respect to removing the lock file in a running system: Don't

The lock file is what allows us to coordinate access to the realm file itself among multiple Core DB objects (realm-core v6 and onwards) or multiple SharedGroup objects (pre realm-core v6). Removing the lock file is guaranteed NOT to work, as soon as multiple such objects need to coordinate access.

Realm does not guarantee any particular coupling between processes/threads and underlying use of DB/SharedGroup objects. We may also want to change any such coupling in the future, just as we've changed it recently. If you employ a scheme which removes or tinkers with the lock file while the app is running, your code may break with next version of Realm. Or it may break with the current version - only you haven't noticed yet :-O

Point is: You do not have control over when multiple DB objects are in play. Sorry.

Don't mess with the lock file.

Totally Agree. And what do you think about this walkaround:

Temporarily release the file lock just before the process being suspended, e.g., when enter background state for iOS main app. This will prevent the app being killed by iOS 14. And because of the entire process has been suspended, it is ok to release the file locks from current process. (Am I right? 🤨) Then, we recover those locks when the process awakes again (enter inactive state). I know it's dirty 😟. But what can I do if Apple never fix it? Is there any better idea?

[finnschiermer]

@gongzhang Unfortunately it is also dangerous. The locking is part of a machinery which allows us to re-initialize the lock file at opportune points. Tinkering with the lock could lead us to initialize the file at the wrong time, leading to the same problem as directly deleting the lock file. It may work if you have absolute control of when exactly you release and reacquire the lock with respect to calls to DB::open() or close() in realm-core. It sounds brittle.

But we're working on a fix. We're not waiting for Apple. I don't have a timeline at the moment.

[gongzhang]

@finnschiermer Thank you for your clear response :) It looks like quite a bit of code in realm-core needs to be carefully modified. I'll stop hacking it for now 😂.

[LorienMan]

For example GRDB suspends database in this case that causes all the transactions that are running at the moment app is being suspended to be failed with an error (and it seems that the transaction can be restarted when the app returns from suspended state). Pretty fair.

[finnschiermer]

@LorienMan Thx for info. That could also be a way forward. It would have to be hand in hand with complete re-initialization of the app, I guess, since our concept of "live" transactions could not survive. But perhaps that's the case anyway. We'll evaluate it against/along with the fix we're currently working on.

[bguidolim]

I just want to drop a friendly reminder here, we should not forget about transactions in background, like changes via background fetching or silent push notifications.

[gongzhang]

I just want to drop a friendly reminder here, we should not forget about transactions in background, like changes via background fetching or silent push notifications.

Exactly. We need to note that code can also be executed when the app is in background. Here we should discuss the suspended state. This is the state that triggers the bug we are discussing here.

(See: iOS App Life Cycle)

[SquaredTiki]

Doesn't appear that Apple has any interest in fixing this for the sakes of binary compatibility, received the following response on my bug report (FB8128103):

This is an issue specific to a third-party, not an Apple issue.

This is a Realm bug that they’re tracking:

https://github.com/realm/realm-cocoa/issues/6671

Please contact Realm for further support.

Please close your feedback report, or let us know if this is still an issue for you.

[gongzhang]

Hey guys, I just reproduce the bug without Realm. It's all about holding flock in App Group directory. You can follow these steps to quickly reproduce the weird crash:

1. Create a new iOS App project (Objective-C) using Xcode.

2. Add an App Group container in project setting page.

3. Paste this code snippet in application:didFinishLaunchingWithOptions::

    // 1. prepare a non-empty file under App Group directory

    NSFileManager* fileManager = [NSFileManager defaultManager];

    NSURL* dir = [fileManager containerURLForSecurityApplicationGroupIdentifier:@"group.xxx.xxx....."]; <= your group id here

    NSURL* fileUrl = [dir URLByAppendingPathComponent:@"file"];

    [fileManager createFileAtPath:[fileUrl path]

                         contents:[@"some data..." dataUsingEncoding:(NSUTF8StringEncoding)]

                       attributes:nil];

    // 2. hold a file lock

    int fd = open([fileUrl path].UTF8String, O_RDWR);

    int ret = flock(fd, LOCK_SH);

4. Debug the project on a physical device running iOS 14 b3/b4/b5.

5. The app will be killed after you return to home screen.

6. If you unlock the file by calling flock(fd, LOCK_UN) before the app enters suspended state, the app won't be killed by iOS.

---

Note that:

• This only crash on a physical device, not a simulator.

• Xcode does not handle it like a normal crash. It just print a termination message in console and ends the debug session gracefully.

@SquaredTiki 🤪Just reply them with this code. Even this is the new expected behavior of iOS 14, they still need to fix the last two bullets I mentioned.

[kunalsood]

Automatically closed because the pull request that fixes this has been merged into realm-core? So, we just need to wait for someone to upgrade core to the version with the fix on realm-cocoa master branch?

[finnschiermer]

..and a Core release with the fix...

[kunalsood]

..and a Core release with the fix...

Excellent! Thanks so much!

[robbiet480]

@finnschiermer How long until a Cocoa update you think?

[robbiet480]

@tgoyne Just opened a PR to bump the core version at https://github.com/realm/realm-cocoa/pull/6722

[marciogranzotto]

I tried to use the branch from #6722 but it still throws a Message from debugger: Terminated due to signal 9. If I run my app with my mock database, without using Realm, that error never happens.

[bguidolim]

iOS 14 b7 seems to solve this issue.

[owenzhao]

iOS 14 b7 seems to solve this issue.

You should aware that Realm 5.3.6 fixed this issue. Just see the release note.

[gongzhang]

Confirmed iOS 14b7 has fixed the issue. This is the final prove this is an apple bug.😂

[tgoyne]

I guess we can disable our workaround? Of course they'd fix it as soon as we got something working...

[ianpward]

😂

[SquaredTiki]

Since the changes to resolve this issue we've started seeing our app freeze (sometimes causing the whole of iOS to freeze) when foregrounded on iOS 10, 11, 12. This was briefly mentioned by @thodang888 in #6722 and I've raised a new issue #6749 for this.