Getting exception on DB deletion

[navaneet]

Hi, I am getting the following exception while deleting a database.

BUG IN CLIENT OF libsqlite3.dylib: database integrity compromised by API violation: vnode unlinked while in use: /Users/nav/Library/Developer/CoreSimulator/Devices/F3E6707C-845A-44D4-9138-36FF2A4A58ED/data/Containers/Data/Application/26F2FF48-C5A8-4F6B-9E9E-BDDF02EF28A4/Documents/databases/flight-hv0999-20180426-ivl-ssa_extensions/com.cloudant.sync.query/indexes.sqlite

Is it because the db is currently in use? Is there any way in the library to check if a db is open and maybe queue the deletion until the file handler is closed?

I have been able to reproduce the issue with the following code.

            let dbName = "flight"
            let DATABASE_PATH = "databases"
            // Check if datastore manager doen't exist
            let path = FileManager.default.documentDirectoryWith(lastPath: DATABASE_PATH)
            var manager: CDTDatastoreManager?
            do {
            manager = try CDTDatastoreManager(directory: path)
            } catch {
                print("\(error)")
            }

            var datastore: CDTDatastore?

            //create new datastore
            do {
                datastore = try manager?.datastoreNamed(dbName)
            } catch {
                print("\(error)")
            }
            datastore?.ensureIndexed(["type", "number", "name"], withName: dbName)

            //insert documents
            for number in 1...10 {
                let rev: CDTDocumentRevision = CDTDocumentRevision(docId: String(number))
                rev.body = ["type":"temp", "number": number, "name": "abc\(number)"]
                do {
                    try datastore?.createDocument(from: rev)
                } catch {
                    print("Error while creating doc \(error)")
                }
            }

            //update indexes
            print("updating all indexes")
            datastore?.updateAllIndexes()

            //explicitly setting datastore to nil so that no more leftover refs
            datastore = nil

            //delete any previous datastore
            do {
                try manager?.deleteDatastoreNamed(dbName)
            } catch {
                print("\(error)")
            }

Here I create a db named flight, then I insert some documents, then I delete the db and I get the below exception in the console

updating all indexes
2018-05-07 16:28:34.057144+0530 Sample[11398:403848] [logging] BUG IN CLIENT OF libsqlite3.dylib: database integrity compromised by API violation: vnode unlinked while in use: /Users/nav/Library/Developer/CoreSimulator/Devices/F3E6707C-845A-44D4-9138-36FF2A4A58ED/data/Containers/Data/Application/3FA97274-3184-4801-B677-631217E1DFAB/Documents/databases/flight_extensions/com.cloudant.sync.query/indexes.sqlite
2018-05-07 16:28:34.058440+0530 Sample[11398:403848] [logging] invalidated open fd: 6 (0x11)

[ricellis]

Is it because the db is currently in use?

Yes, this type of exception is indicative of a sqlite db being deleted while it is still in use. In this case it is the indexes.sqlite db, which is part of the query extension. I believe that, similarly to the datastore db, reference counting is used internally to close the indexes database when no further references are held.

Is there any way in the library to check if a db is open and maybe queue the deletion until the file handler is closed?

Not as it stands, no. As mentioned above the database "close" is handled internally and IIRC the indexing operations are synchronous, so it shouldn't be necessary to check here if the indexes DB is open. For debugging this issue you could try to find where references to the CDTQIndexManager are being held when you try to delete, which could help us find the root cause.

[navaneet]

We are not using CDTQIndexManager directly as you can see in the example I provided, wherein we use the new api for indexing. You should be able to reproduce the issue by running the example code directly. There might be a reference count issue however it's not at the example level but at the library level as there are no strong references explicitly maintained in the example code.

The example by the way is written in swift and uses the latest version of cdtdatastore.

[navaneet]

@ricellis Are there any updates on this issue? Have you been able to reproduce the issue with the provided example?

[ricellis]

Updates will be posted in the issue when there are any. I have not had the opportunity to try and reproduce the problem yet. We will continue to investigate as soon as we are able to.

[ricellis]

I have been able to reproduce the issue, my initial assessment is that the extensions DBs are not closed before the filesystem delete is issued. We'll keep looking into it and start working on a fix.

[navaneet]

Ok thanks for the timely response

[navaneet]

@ricellis It will also be helpful for us if there is an event raised just before db deletion which might help us close of any ui elements and such that might be using the datastore to be deleted. A simple willDeleteDatastore(name: String) event should suffice.

[ricellis]

There are currently no datastore lifecycle events in CDTDatastore, only for changes within a datastore. The datastore deletion itself is synchronous so it should be straightforward to issue an event of your own or perform the actions in your application aligned with your call to deleteDatastoreNamed(dbName).

[ricellis]

Closing as #428 has merged, fix will be in the next release.

[tomblench]

2.0.3 released, please try it and open a new issue if you are still having problems.

[navaneet]

K let me try it out, thanks

[navaneet]

@tomblench @ricellis Hi, I am still able to reproduce both the issues I reported with 2.0.3. Please try with the sample code I provided above. I don't see any files except theCDTDatastore/Version.h file being updated in https://github.com/cloudant/CDTDatastore/releases which has just one commit