Lightweight migration and initializing CoreStore

[tosbaha]

Hi, I decided to finally move my statck to CoreStore but I have questions about initializing and doing lightweight migration. I upgraded my model from Kargo to KargoV2 and coded NSEntityMigrationPolicy as I said in issue #142. With my old CoreData code and using

[NSInferMappingModelAutomaticallyOption: true, 
NSMigratePersistentStoresAutomaticallyOption: true]

update works without a problem. I also use different URL for sqlite. I have come up with below code and I appreciate if you can tell me whether it is good or not. I also didn't understand the usage of migrationProcess. What we are doing with let migrationProcess ?

        let dataStack = DataStack(xcodeModelName: "Kargo",migrationChain: ["Kargo","KargoV2"])
        let sqliteFileURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: "group.com.myapp")!.appendingPathComponent("Kargo.sqlite")
        let migrationProcess = dataStack.addStorage(
            SQLiteStore(
                fileURL: sqliteFileURL, // set the target file URL for the sqlite file
                localStorageOptions: .recreateStoreOnModelMismatch // if migration paths cannot be resolved, recreate the sqlite file
            ),
            completion: { (result) -> Void in
                switch result {
                case .success(let storage):
                    print("Successfully added sqlite store: \(storage)")
                case .failure(let error):
                    print("Failed adding sqlite store with error: \(error)")
                }
            }
        )

        CoreStore.defaultStack = dataStack

[JohnEstropia]

@tosbaha Your code looks okay.

What we are doing with let migrationProcess ?

It's actually migrationProgress, and it's an instance of a Progress (NSProgress). If it's nil it means your store is in the latest version and doesn't need an update. Otherwise you can use that Progress instance to track the progress of migrations.

[tosbaha]

Thanks for the fast reply. I don't need to track to migration process. Since I am doing lightweight migration, is it better to use dataStack.addStorage or dataStack.addStorageAndWait ? AFAIK, using asynchronous call helps to not blocking the UI. Library I guess uses ListMonitor abstraction for NSFetchedResultsController. Will there be any problem if I use synchronous or asynchronous version?

[tosbaha]

I tried above code but gives bunch of errors and warnings. I am not sure why it is giving me those. Here is the code and Log.

            let dataStack = DataStack(xcodeModelName: "Kargo",migrationChain: ["Kargo","KargoV2"])
            let sqliteFileURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: "group.com.myapp")!.appendingPathComponent("Kargo.sqlite")

        do {
            try dataStack.addStorageAndWait(
                SQLiteStore(
                    fileURL: sqliteFileURL, // set the target file URL for the sqlite file
                    localStorageOptions: .recreateStoreOnModelMismatch // if migration paths cannot be resolved, recreate the sqlite file
                )
            )

            CoreStore.defaultStack = dataStack

        } catch {
            print("Error coredata \(error)")
        }

Logs

⚠️ [CoreStore: Warning] XcodeDataModelSchema.swift:79 from(modelName:bundle:migrationChain:)
  ↪︎ The 'MigrationChain' leaf versions do not include the model file's current version. Resolving to version "KargoV2".

2018-04-25 00:17:45.534513+0300 Kargotakip[1171:94823] [error] error: -addPersistentStoreWithType:SQLite configuration:(null) URL:file:///Users/mustafa/Library/Developer/CoreSimulator/Devices/25BB426F-1331-4C43-83A3-F8343512CACD/data/Containers/Shared/AppGroup/AA1DC7FD-081A-458E-A3EA-DAD54314A3D4/Kargo.sqlite options:{
    NSSQLitePragmasOption =     {
        "journal_mode" = WAL;
    };
} ... returned error Error Domain=NSCocoaErrorDomain Code=134100 "(null)" UserInfo={metadata={
    NSPersistenceFrameworkVersion = 640;
    NSStoreModelVersionHashes =     {
        KargoCD = <7c895050 9332d7c5 e7242e67 9487226d 648cb8fe d7ef0155 b2c21365 700d1f56>;
        KargoDetailCD = <773f65ec 83377b36 046ec385 8967bbf3 f34c7510 0245b58c 4c59d41b 2a800d58>;
    };
    NSStoreModelVersionHashesVersion = 3;
    NSStoreModelVersionIdentifiers =     (
        ""
    );
    NSStoreType = SQLite;
    NSStoreUUID = "8E3ADB60-D46E-466A-91C6-840A45557571";
    "_NSAutoVacuumLevel" = 2;
}, reason=The model used to open the store is incompatible with the one used to create the store} with userInfo dictionary {
    metadata =     {
        NSPersistenceFrameworkVersion = 640;
        NSStoreModelVersionHashes =         {
            KargoCD = <7c895050 9332d7c5 e7242e67 9487226d 648cb8fe d7ef0155 b2c21365 700d1f56>;
            KargoDetailCD = <773f65ec 83377b36 046ec385 8967bbf3 f34c7510 0245b58c 4c59d41b 2a800d58>;
        };
        NSStoreModelVersionHashesVersion = 3;
        NSStoreModelVersionIdentifiers =         (
            ""
        );
        NSStoreType = SQLite;
        NSStoreUUID = "8E3ADB60-D46E-466A-91C6-840A45557571";
        "_NSAutoVacuumLevel" = 2;
    };
    reason = "The model used to open the store is incompatible with the one used to create the store";
}
CoreData: error: -addPersistentStoreWithType:SQLite configuration:(null) URL:file:///Users/mustafa/Library/Developer/CoreSimulator/Devices/25BB426F-1331-4C43-83A3-F8343512CACD/data/Containers/Shared/AppGroup/AA1DC7FD-081A-458E-A3EA-DAD54314A3D4/Kargo.sqlite options:{
    NSSQLitePragmasOption =     {
        "journal_mode" = WAL;
    };
} ... returned error Error Domain=NSCocoaErrorDomain Code=134100 "(null)" UserInfo={metadata={
    NSPersistenceFrameworkVersion = 640;
    NSStoreModelVersionHashes =     {
        KargoCD = <7c895050 9332d7c5 e7242e67 9487226d 648cb8fe d7ef0155 b2c21365 700d1f56>;
        KargoDetailCD = <773f65ec 83377b36 046ec385 8967bbf3 f34c7510 0245b58c 4c59d41b 2a800d58>;
    };
    NSStoreModelVersionHashesVersion = 3;
    NSStoreModelVersionIdentifiers =     (
        ""
    );
    NSStoreType = SQLite;
    NSStoreUUID = "8E3ADB60-D46E-466A-91C6-840A45557571";
    "_NSAutoVacuumLevel" = 2;
}, reason=The model used to open the store is incompatible with the one used to create the store} with userInfo dictionary {
    metadata =     {
        NSPersistenceFrameworkVersion = 640;
        NSStoreModelVersionHashes =         {
            KargoCD = <7c895050 9332d7c5 e7242e67 9487226d 648cb8fe d7ef0155 b2c21365 700d1f56>;
            KargoDetailCD = <773f65ec 83377b36 046ec385 8967bbf3 f34c7510 0245b58c 4c59d41b 2a800d58>;
        };
        NSStoreModelVersionHashesVersion = 3;
        NSStoreModelVersionIdentifiers =         (
            ""
        );
        NSStoreType = SQLite;
        NSStoreUUID = "8E3ADB60-D46E-466A-91C6-840A45557571";
        "_NSAutoVacuumLevel" = 2;
    };
    reason = "The model used to open the store is incompatible with the one used to create the store";
}
2018-04-25 00:17:45.607471+0300 Kargotakip[1171:94875] [logging] BUG IN CLIENT OF libsqlite3.dylib: database integrity compromised by API violation: vnode unlinked while in use: /Users/mustafa/Library/Developer/CoreSimulator/Devices/25BB426F-1331-4C43-83A3-F8343512CACD/data/Containers/Shared/AppGroup/AA1DC7FD-081A-458E-A3EA-DAD54314A3D4/Kargo.sqlite-wal
2018-04-25 00:17:45.607674+0300 Kargotakip[1171:94875] [logging] invalidated open fd: 6 (0x11)

FINAL EDIT: I solved migration problem with using .allowSynchronousLightweightMigration flag. However last error related to API violation is related to iOS 11 I guess. I searched and found similar issue in Ensembles framework. It seems in iOS 11, we should also add NSBinaryStoreInsecureDecodingCompatibilityOption to true. I checked the code base and it seems this is not set by library. Here is PR that is merged to Ensembles. https://github.com/drewmccormack/ensembles/pull/260/files

Apple forum link https://forums.developer.apple.com/thread/84588

[JohnEstropia]

@tosbaha Thanks for pointing that out, I'll check the details of that issue.

[PraLeshik]

Hello. In call dataStack.addStorageAndWait we put one storage. Is it source or destination storage? I assume we have to use both ones in call. And I have the same problem with "The model used to open the store is incompatible with the one used to create the store" during migration.

[JohnEstropia]

@PraLeshik You need to give the new model to addStorage. You also need to make sure that you are creating the DataStack using a CoreStoreSchema that contains all your old models or Core Data cannot infer your migrations.

[PraLeshik]

Thanks, @JohnEstropia Not sure if I use storage initialization in a right way. Still getting error:

[CoreStore: Fatal Error] BaseDataTransaction.swift:83 create ↪︎ Attempted to create an entity of type 'UserProfile', but a destination persistent store containing the entity type could not be found.

Here is my code:

        let dataStack = DataStack(DatabaseSchema.v2.schema) // prev. version was v1

        do {
            try dataStack.addStorageAndWait(SQLiteStore())
        } catch {
            print("<DatabaseManager> SQLiteStore initialization error: \(error.localizedDescription)")
        }

        CoreStore.defaultStack = dataStack
.....

enum DatabaseSchema {
    case v1
    case v2

    public var modelVersion: String {
        switch self {
        case .v1:
            return "v1"
        case .v2:
            return "v2"
        }
    }

    public var schema: CoreStoreSchema {
        switch self {
        case .v1:
            return CoreStoreSchema(
                modelVersion: self.modelVersion,
                entities: [
                    Entity<UserProfile>("UserProfile"),
                    Entity<ChildProfile>("ChildProfile"),
                    Entity<EventModel>("Event", isAbstract: true),
                    Entity<BreastEvent>("BreastEvent"),
                    Entity<BottleEvent>("BottleEvent"),
                    Entity<SolidEvent>("SolidEvent"),
                    Entity<Ingredient>("Ingredient")
                ]
            )

        case .v2:
            return CoreStoreSchema(
                modelVersion: self.modelVersion,
                entities: [
                    Entity<UserProfile>("UserProfile"),
                    Entity<ChildProfile>("ChildProfile"),
                    Entity<EventModel>("Event", isAbstract: true),
                    Entity<BreastEvent>("BreastEvent"),
                    Entity<BottleEvent>("BottleEvent"),
                    Entity<SolidEvent>("SolidEvent"),
                    Entity<Ingredient>("Ingredient"),
                    Entity<PumpingEvent>("PumpingEvent") // new model, other ones are the same as in v1 schema
                ]
            )
        }
    }
}

[JohnEstropia]

@PraLeshik You'll need to pass BOTH versions to the DataStack. (See: SchemaHistory)

CoreStore.defaultStack = DataStack(
    schemaHistory: SchemaHistory(
        allSchema: [DatabaseSchema.v1.schema, DatabaseSchema.v2.schema]
    )
)

As for your entities, you mentioned that other than PumpingEvent all other entities are the same. Just make sure that you have no relationship changes as well (this includes any entity with relationship to PumpingEvent). If you have any relationship changes, you will need to duplicate those related classes as well.

[PraLeshik]

@JohnEstropia Thanks for your answer. Still getting error "Attempted to create an entity of type 'UserProfile', but a destination persistent store containing the entity type could not be found.". Is there something wrong in storage initialization?

        let schemaHistory = SchemaHistory(allSchema: [DatabaseSchema.v1.schema, DatabaseSchema.v2.schema], migrationChain: ["v1", "v2"])
        CoreStore.defaultStack = DataStack(schemaHistory:schemaHistory)

        do {
            let localStorage = SQLiteStore(fileURL: databaseURL, localStorageOptions: .allowSynchronousLightweightMigration)
            try CoreStore.defaultStack.addStorageAndWait(localStorage)
        } catch {
            print("SQLiteStore initialization error: \(error.localizedDescription)")
        }