search

groue / GRDB.swift

A toolkit for SQLite databases, with a focus on application development
MIT License
6.73k stars 697 forks source link

Migrating existing DB to an AppGroup #643

Closed foxware00 closed 4 years ago

foxware00 commented 4 years ago

Question

I'm trying to create a Share extension that has access to my host application's database. The problem is that users already have the SQLite file created and populated. I know the path of the existing database and where the new one is.

My question is there a clean and easy way to migrate the SQLite file to another path in a one-off operation. I'm guessing there are a few files that all need to be copied and duplicated cleanly.

My initial thoughts would be to open the current DB, create a new on in a new path and copy all the data over. This might take a few seconds the first run but seems like the cleanest way to do it. I'd delete the old DB, and switch over to the new DB once the migration has completed. Or would a quicker direct file copy make more sense?

I've found a CoreData method that migrates data from one URL to another, is this something that's easy to achieve in GRDB?

Environment

GRDB flavor(s): GRDB GRDB version: 4.1.1 Installation method: CocoaPods Xcode version: 11.1 Swift version: 5.1 Platform(s) running GRDB: iOS macOS version running Xcode: 10.14.5

groue commented 4 years ago

The blog post above contains this interesting paragraph:

Due to incorrectly placed task markers, [...] iOS would unceremoniously kill the process — making it look like it had crashed somewhere inside of Core Data..

I spent several days reworking a bunch of old code to ensure that task markers were started and ended in the proper sequence, and to ensure that the task wasn’t marked as complete until after the entire process was done.

One could not have expressed it in a better way. Having GRDB users tell how they spent several days dealing with markers is exactly what I want to avoid. What a shit work.

No. What I want is something like:

do {
    try databaseStuff()
} catch {
    // Handle eventual error due to lock prevention.
    // For example try again when database becomes available again.
}

We see that we'll have to expose this notion of "database becomes available again". In spirit it should be very close to UIApplicationProtectedDataDidBecomeAvailable.

In the same trend of thought, application developers will also have to be able to easily restart database observations that have failed because of lock prevention (and see how it fits with RxGRDB and GRDBCombine).

There's still a lot of work before we can ship the "App Group edition" of GRDB. And I'm not talking about cross-process database observation (this will probably ship in the "App Group Gold edition").

foxware00 commented 4 years ago

https://blog.iconfactory.com/2019/08/the-curious-case-of-the-core-data-crash/

This was exactly my path. Nice to know I wasn't alone.

We see that we'll have to expose this notion of "database becomes available again". In spirit it should be very close to UIApplicationProtectedDataDidBecomeAvailable.

Does this effectively notify us to unblock the DB?

The real problem here is basically that the ONLY time it's truly safe to start a background task is in EXACTLY the same runloop cycle as your app was woken by the system. In any other situation (for example, in a different thread or at some later point after you were woken), there isn't any way to guarantee that your app hasn't already started to suspend.

It sounds like we might need to live with the occasional crash. Minimising the potential impact of such. It seems like a scenario Apple has just decided to accept the fact there they might kill you at any point. An interesting stance indeed.

Interestingly in my scenario the second the app opens from didReceiveRemoteNotification or performFetchWithCompletionHandler I request a background task, only ending it when completed successfully. Obviously, it still crashes because I don't actually forcefully stop the database execution. I need to interrupt the database manually, something I need to try. Clearly this isn't the solution for everyone and is exactly what we'd ideally resolve with the handling being internal in the clean clear API you suggest. I plan on adding further background processing in the near future which would only increase the surface area of this, ideally, unnecessary work. "shit work" indeed.

I feel your pain and frustration and thank your efforts in rolling out a robust solution for the rest of us and to keep GRDB simple and effective API.

groue commented 4 years ago

The "final" PR is drafted: #663

foxware00 commented 4 years ago

The "final" PR is drafted: #663

Very exciting! I'll have a proper read through shortly

groue commented 4 years ago

663 is able to prevent 0xdead10cc for applications, but I was not able yet to work on extensions, mainly because I don't know how to reproduce the crash in an extension. It's obviously impossible to try to prevent a crash that is impossible to reproduce.

663 is thus on hold, but I want to merge into master the preliminary work. This is #668, flagged experimental.

groue commented 4 years ago

If you know how to reproduce the 0xdead10cc in an extension, please give your recipe!

foxware00 commented 4 years ago

@groue unfortunately I haven't experienced an extension causing 0xdead10cc. I was only seeing issues when dealing with the main app doing background fetches.

groue commented 4 years ago

There are some evidence that this crash exists in extensions, such as this tweet by @marcoarment. But it's difficult to grab any actionable information. If shipping a real app is a mandatory step, then we'll hardly see any advance on this topic in an open source project.

foxware00 commented 4 years ago

Agreed, I saw this too. From my testing it's less predictable from an app extension. What have you tried so far? In the same way your tests hold a lock on the database during suspsension. Would simply holding a lock, then calling through to self.extensionContext?.completeRequest(returningItems: [], completionHandler: nil) trigger the same scenario. Or is this what you've already tried?

I can try test this later on today.

groue commented 4 years ago

Agreed, I saw this too. From my testing it's less predictable from an app extension. What have you tried so far?

Would simply holding a lock, then calling through to self.extensionContext?.completeRequest(returningItems: [], completionHandler: nil) trigger the same scenario.

I don't understand. What do you have in mind?

foxware00 commented 4 years ago

Ran a sharing extension on the device. Keep a transaction open forever.

Calling self.extensionContext?.completeRequest(returningItems: [], completionHandler: nil) from the share extension closes the extension and can hand off the URL session to the containing app. My thinking way transactions held here would cause the error. It's very odd that you're seeing status code 0, when it sounds like it should almost definitely be killing the process to clear the lock.

groue commented 4 years ago

It's very odd that you're seeing status code 0, when it sounds like it should almost definitely be killing the process to clear the lock.

Yes. It could also be a side-effect of the debugger. But when I run the extension without any debugger attached, I don't see anything weird in Console.app. The Devices window of Xcode does not show any crash log.

Of course, I may just have missed something. This is a ~chore~ very long process 😅

Calling self.extensionContext?.completeRequest(returningItems: [], completionHandler: nil) from the share extension closes the extension and can hand off the URL session to the containing app. My thinking way transactions held here would cause the error.

You mean the transaction held by the extension. Thanks for the tip, this may give something. Did you make an attempt?

foxware00 commented 4 years ago

One moment, let me give it a go...

Bingo. Is this what you're expecting?

default 14:03:13.170361 +0000   runningboardd   [xpcservice<com.companyname.app.AppNameShare>:740] Suspending task.
default 14:03:13.170482 +0000   runningboardd   Calculated state for xpcservice<com.companyname.app.AppNameShare>: running-suspended (role: None)
error   14:03:13.172627 +0000   runningboardd   not an SQLite database: /var/mobile/Containers/Data/PluginKitPlugin/6C2AFE22-6746-4EC4-B778-14AFB51721D4/Library/Caches/com.crashlytics.data/com.companyname.app.AppNameShare/v3/active/914c3246306d43ea9e8e17bad2b332e7/sdk.log
default 14:03:13.174155 +0000   mediaserverd    -CMSessionMgr- CMSessionMgrHandleApplicationStateChange: CMSession: Client com.companyname.app.AppNameShare with pid '740' is now Background Suspended. Background entitlement: NO ActiveLongFormVideoSession: NO WhitelistedLongFormVideoApp NO
default 14:03:13.186477 +0000   mediaserverd    -CMSessionMgr- CMSessionMgrHandleApplicationStateChange: CMSession: Sending stop command to com.companyname.app.AppNameShare with pid '740' because client is background suspended and there is no AirPlay video session for it
error   14:03:13.187505 +0000   runningboardd   [xpcservice<com.companyname.app.AppNameShare>:740] suspended with locked system files:
/var/mobile/Containers/Shared/AppGroup/635520FC-602F-4747-84AA-67E18F0E1ECA/appName-db.sqlite
not in allowed directories:
(null)
default 14:03:13.187558 +0000   runningboardd   [xpcservice<com.companyname.app.AppNameShare>:740] Terminating with context: <RBSTerminateContext: 0x104acf260; domain: 15; code: 0xDEAD10CC; explanation: "[xpcservice<com.companyname.app.AppNameShare>:740] was suspended with locked system files:
/var/mobile/Containers/Shared/AppGroup/635520FC-602F-4747-84AA-67E18F0E1ECA/appName-db.sqlite
not in allowed directories:
(null)"; reportType: 1; maxRole: 7; maxTerminationResistance: 3>
default 14:03:13.187593 +0000   runningboardd   [xpcservice<com.companyname.app.AppNameShare>:740] terminate_with_reason() success
default 14:03:13.187699 +0000   runningboardd   [xpcservice<com.companyname.app.AppNameShare>:740] Set darwin role to: None
default 14:03:13.187903 +0000   backboardd  Connection removed: IOHIDEventSystemConnection uuid:7C87D6B9-C72E-4D83-A850-46B488DB4552 pid:740 process:AppName type:Passive entitlements:0x0 caller:BackBoardServices: <redacted> + 380 attributes:{
    HighFrequency = 0;
    bundleID = "com.companyname.app.AppNameShare";
    pid = 740;
} inactive:0 events:8 mask:0x800
default 14:03:13.188821 +0000   SpringBoard [xpcservice<com.companyname.app.AppNameShare>:740] Now flagged as pending exit for reason: workspace client connection invalidated
default 14:03:13.191227 +0000   runningboardd   XPC connection invalidated: [xpcservice<com.companyname.app.AppNameShare>:740]
default 14:03:13.192977 +0000   runningboardd   [xpcservice<com.companyname.app.AppNameShare>:740] Death sentinel fired!
default 14:03:13.195519 +0000   itunesstored    Updating configuration of monitor <RBSProcessMonitorConfiguration: 0x102cd8eb0; id: M139-1; qos: 17> {
    predicates = {
        <RBSProcessPredicate: 0x102cfb0c0> {
            predicate = <RBSCompoundPredicate; <RBSProcessBundleIdentifierPredicate; com.supersmashing.ios.calderstewart>; <RBSCompoundPredicate; <RBSProcessEUIDPredicate; 501>; <RBSProcessBKSLegacyPredicate: 0x102c427f0>>>;
        };
    }
    descriptor = <RBSProcessStateDescriptor: 0x102cf7690; values: 11> {
        namespaces = {
            com.apple.frontboard.visibility;
        }
    };
}
error   14:03:13.231278 +0000   runningboardd   RBSStateCapture remove item called for untracked item <RBProcessMonitorObserver: 0x104947a50; <RBProcess: 0x10494baa0; 740; identity: xpcservice<com.companyname.app.AppNameShare>>; configCount: 0>
default 14:03:13.274739 +0000   symptomsd   defusing ticker tickerFatal having seen progress by flow for com.companyname.app, rxbytes 6976 duration 7.951 seconds started at time: Thu Dec 12 14:03:05 2019
default 14:03:13.296903 +0000   runningboardd   Removing process: [xpcservice<com.companyname.app.AppNameShare>:740]
default 14:03:13.297181 +0000   runningboardd   Removing assertions for terminated process: [xpcservice<com.companyname.app.AppNameShare>:740]
default 14:03:13.297851 +0000   runningboardd   Calculated state for xpcservice<com.companyname.app.AppNameShare>: none (role: None)
default 14:03:13.304488 +0000   SpringBoard Firing exit handlers for 740 with context <RBSProcessExitContext; unknown; terminationContext: <RBSTerminateContext: 0x28036bf40; domain: 15; code: 0xDEAD10CC; explanation: "[xpcservice<com.companyname.app.AppNameShare>:740] was suspended with locked system files:
/var/mobile/Containers/Shared/AppGroup/635520FC-602F-4747-84AA-67E18F0E1ECA/appName-db.sqlite
not in allowed directories:
(null)"; reportType: 1; maxRole: 7; maxTerminationResistance: 3>>
default 14:03:13.304572 +0000   SpringBoard [xpcservice<com.companyname.app.AppNameShare>:740] Process exited: <RBSProcessExitContext; unknown; terminationContext: <RBSTerminateContext: 0x28036bf40; domain: 15; code: 0xDEAD10CC; explanation: "[xpcservice<com.companyname.app.AppNameShare>:740] was suspended with locked system files:
/var/mobile/Containers/Shared/AppGroup/635520FC-602F-4747-84AA-67E18F0E1ECA/appName-db.sqlite
not in allowed directories:
(null)"; reportType: 1; maxRole: 7; maxTerminationResistance: 3>>.
default 14:03:13.304623 +0000   SpringBoard [xpcservice<com.companyname.app.AppNameShare>:740] Setting process task state to: Not Running
default 14:03:13.305487 +0000   runningboardd   Calculated state for xpcservice<com.companyname.app.AppNameShare>: none (role: None)

Looks like it worked

terminationContext: <RBSTerminateContext: 0x28036bf40; domain: 15; code: 0xDEAD10CC; explanation: "[xpcservice<com.companyname.app.AppNameShare>:740] was suspended with locked system files

To replicate this. I installed the application. Removed the debugger and opened console

This is code that caused it. I added this to a button press within the ShareExtension. The openLongRunningTransaction is stolen from your 10deadcc test repo

Database.shared.openLongRunningTransaction { result in 
    print("transaction result: \(result)")
}
self.extensionContext?.completeRequest(returningItems: [], completionHandler: nil)
groue commented 4 years ago

Thanks @foxware00. Meanwhile, I'll shortly merge and ship #668. It does not automatically suspend databases, but you can do it from your app and extension. You are warmly encouraged to perform your experiments and see if you can get rid of 0xdead10cc. And profit from the other shared database setup advice.

foxware00 commented 4 years ago

@groue Thank you I will shortly have a play. Thanks for all your hard work on the topic. Getting closer to a really fantastic and robust solution.