search

google / android-fhir

The Android FHIR SDK is a set of Kotlin libraries for building offline-capable, mobile-first healthcare applications using the HL7® FHIR® standard on Android.
https://google.github.io/android-fhir/
Apache License 2.0
494 stars 295 forks source link

Database migration exception force closes the app #2170

Closed ellykits closed 1 year ago

ellykits commented 1 year ago

Describe the bug Migration exception causing app crash

Component Core library

To Reproduce Steps to reproduce the behavior: Upgrade to the latest version of SDK. Run an existing application. Experience application crash.

Expected behavior Database Migration should run successfully without causing any app crash.

Additional context Logs

FATAL EXCEPTION: DefaultDispatcher-worker-1
                                                                                                    Process: org.smartregister.opensrp.ecbis, PID: 11462
                                                                                                    java.lang.IllegalStateException: Migration didn't properly
                                                                                                    handle: TokenIndexEntity(com.google.android.fhir.db.impl.entities.TokenIndexEntity).
                                                                                                     Expected:
                                                                                                    TableInfo{name='TokenIndexEntity', columns={index_path=Column{name='index_path', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}, index_value=Column{name='index_value', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}, index_system=Column{name='index_system', type='TEXT', affinity='2', notNull=false, primaryKeyPosition=0, defaultValue='undefined'}, id=Column{name='id', type='INTEGER', affinity='3', notNull=true, primaryKeyPosition=1, defaultValue='undefined'}, index_name=Column{name='index_name', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}, resourceUuid=Column{name='resourceUuid', type='BLOB', affinity='5', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}, resourceType=Column{name='resourceType', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}}, foreignKeys=[ForeignKey{referenceTable='ResourceEntity', onDelete='CASCADE
                                                                                                    +', onUpdate='NO
                                                                                                    ACTION', columnNames=[resourceUuid], referenceColumnNames=[resourceUuid]}], indices=[Index{name='index_TokenIndexEntity_resourceType_index_name_index_system_index_value_resourceUuid', unique=false, columns=[resourceType, index_name, index_system, index_value, resourceUuid], orders=[ASC, ASC, ASC, ASC, ASC]'}, Index{name='index_TokenIndexEntity_resourceUuid', unique=false, columns=[resourceUuid], orders=[ASC]'}]}
                                                                                                     Found:
                                                                                                    TableInfo{name='TokenIndexEntity', columns={id=Column{name='id', type='INTEGER', affinity='3', notNull=true, primaryKeyPosition=1, defaultValue='undefined'}, resourceUuid=Column{name='resourceUuid', type='BLOB', affinity='5', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}, resourceType=Column{name='resourceType', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}, index_name=Column{name='index_name', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}, index_path=Column{name='index_path', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}, index_system=Column{name='index_system', type='TEXT', affinity='2', notNull=false, primaryKeyPosition=0, defaultValue='undefined'}, index_value=Column{name='index_value', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}}, foreignKeys=[ForeignKey{referenceTable='ResourceEntity', onDelete='CASCADE
                                                                                                    +', onUpdate='NO
                                                                                                    ACTION', columnNames=[resourceUuid], referenceColumnNames=[resourceUuid]}], indices=[Index{name='index_TokenIndexEntity_resourceUuid', unique=false, columns=[resourceUuid], orders=[ASC]'}, Index{name='index_TokenIndexEntity_resourceType_index_name_index_system_index_value', unique=false, columns=[resourceType, index_name, index_system, index_value], orders=[ASC, ASC, ASC, ASC]'}]}
                                                                                                        at androidx.room.RoomOpenHelper.onUpgrade(RoomOpenHelper.kt:94)
                                                                                                        at androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper$OpenHelper.onUpgrade(FrameworkSQLiteOpenHelper.kt:253)
                                                                                                        at android.database.sqlite.SQLiteOpenHelper.getDatabaseLocked(SQLiteOpenHelper.java:416)
                                                                                                        at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:316)
                                                                                                        at
                                                                                                    androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper$OpenHelper.getWritableOrReadableDatabase(FrameworkSQLiteOpenHelper.kt:232)
                                                                                                        at androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper$OpenHelper.innerGetDatabase(FrameworkSQLiteOpenHelper.kt:190)
                                                                                                        at androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper$OpenHelper.getSupportDatabase(FrameworkSQLiteOpenHelper.kt:151)
                                                                                                        at androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper.getWritableDatabase(FrameworkSQLiteOpenHelper.kt:104)
2023-09-14 14:36:13.260 11462-11494 AndroidRuntime          org.smartregister.opensrp.ecbis      E      at androidx.room.RoomDatabase.internalBeginTransaction(RoomDatabase.kt:528)
                                                                                                        at androidx.room.RoomDatabase.beginTransaction(RoomDatabase.kt:517)
                                                                                                        at androidx.room.RoomDatabaseKt$withTransaction$transactionBlock$1.invokeSuspend(RoomDatabaseExt.kt:54)
                                                                                                        at androidx.room.RoomDatabaseKt$withTransaction$transactionBlock$1.invoke(Unknown Source:8)
                                                                                                        at androidx.room.RoomDatabaseKt$withTransaction$transactionBlock$1.invoke(Unknown Source:4)
                                                                                                        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:78)
                                                                                                        at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:167)
                                                                                                        at kotlinx.coroutines.BuildersKt.withContext(Unknown Source:1)
                                                                                                        at androidx.room.RoomDatabaseKt$startTransactionCoroutine$2$1$1.invokeSuspend(RoomDatabaseExt.kt:97)
                                                                                                        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
                                                                                                        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
                                                                                                        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:280)
                                                                                                        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
                                                                                                        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
                                                                                                        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source:1)
                                                                                                        at androidx.room.RoomDatabaseKt$startTransactionCoroutine$2$1.run(RoomDatabaseExt.kt:93)
                                                                                                        at androidx.room.TransactionExecutor.execute$lambda$1$lambda$0(TransactionExecutor.kt:36)
                                                                                                        at androidx.room.TransactionExecutor.$r8$lambda$AympDHYBb78s7_N_9gRsXF0sHiw(Unknown Source:0)
                                                                                                        at androidx.room.TransactionExecutor$$ExternalSyntheticLambda0.run(Unknown Source:4)
                                                                                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
                                                                                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
                                                                                                        at java.lang.Thread.run(Thread.java:1012)
                                                                                                        Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@32d7b7b, Dispatchers.IO]

I cleaned up the TableInfo logs, did a diff-check, and found an issue in the indices section.

Expected

{
 name='TokenIndexEntity',
 columns={
  id=Column{name='id', type='INTEGER', affinity='3', notNull=true, primaryKeyPosition=1, defaultValue='undefined'},
  resourceUuid=Column{name='resourceUuid', type='BLOB', affinity='5', notNull=true, primaryKeyPosition=0, defaultValue='undefined'},
  resourceType=Column{name='resourceType', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'},
  index_name=Column{name='index_name', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'},
  index_path=Column{name='index_path', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'},
  index_system=Column{name='index_system', type='TEXT', affinity='2', notNull=false, primaryKeyPosition=0, defaultValue='undefined'},
  index_value=Column{name='index_value', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}
 },

 foreignKeys=[ForeignKey{referenceTable='ResourceEntity', onDelete='CASCADE', onUpdate='NO ACTION', columnNames=[resourceUuid], referenceColumnNames=[resourceUuid]}],

 indices=[
  Index{name='index_TokenIndexEntity_resourceUuid', unique=false, columns=[resourceUuid], orders=[ASC]'},
  Index{name='index_TokenIndexEntity_resourceType_index_name_index_system_index_value_resourceUuid', unique=false, columns=[resourceType, index_name, index_system, index_value, resourceUuid], orders=[ASC, ASC, ASC, ASC, ASC]'}
 ]
}

Found

{
 name='TokenIndexEntity',
 columns={
  id=Column{name='id', type='INTEGER', affinity='3', notNull=true, primaryKeyPosition=1, defaultValue='undefined'},
  resourceUuid=Column{name='resourceUuid', type='BLOB', affinity='5', notNull=true, primaryKeyPosition=0, defaultValue='undefined'},
  resourceType=Column{name='resourceType', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'},
  index_name=Column{name='index_name', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'},
  index_path=Column{name='index_path', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'},
  index_system=Column{name='index_system', type='TEXT', affinity='2', notNull=false, primaryKeyPosition=0, defaultValue='undefined'},
  index_value=Column{name='index_value', type='TEXT', affinity='2', notNull=true, primaryKeyPosition=0, defaultValue='undefined'}
 },

 foreignKeys=[ForeignKey{referenceTable='ResourceEntity', onDelete='CASCADE', onUpdate='NO ACTION', columnNames=[resourceUuid], referenceColumnNames=[resourceUuid]}],

 indices=[
  Index{name='index_TokenIndexEntity_resourceUuid', unique=false, columns=[resourceUuid], orders=[ASC]'},
  Index{name='index_TokenIndexEntity_resourceType_index_name_index_system_index_value', unique=false, columns=[resourceType, index_name, index_system, index_value], orders=[ASC, ASC, ASC, ASC]'}
 ]

Would you like to work on the issue? Sure

aditya-07 commented 1 year ago

@ellykits Can you confirm the engine database version of the app before the SDK update? Ideally, MIGRATION_3_4 should have taken care of this. Also, if you can attach any other logs leading to the crash, it would be helpful as well.

ellykits commented 1 year ago

@aditya-07 Apparently, there are no logs other than the one shared. What's the expectation for the migration when the Database version is jumped for instance if the last version is 3 (since the last upgrade) and after the upgrade the new version is 6? Will Room trigger migration one after the other 3 to 4, 4 to 5, and so on?

ellykits commented 1 year ago

Probably the changes that were made in the MIGRATION_3_4 to resolve the issue in TokenIndexEntity table were done after after our app had applied the version 3 migration. There are a couple of code changes in MIGRATION_3_4. This may have occurred in our versioned releases of the engine library.

aditya-07 commented 1 year ago

@aditya-07 Apparently, there are no logs other than the one shared. What's the expectation for the migration when the Database version is jumped for instance if the last version is 3 (since the last upgrade) and after the upgrade the new version is 6? Will Room trigger migration one after the other 3 to 4, 4 to 5, and so on?

Yes, the DB will run all the provided migrations sequentially from the last version to the latest version of the database.

jingtang10 commented 1 year ago

@ellykits can you please dump the db schema for us to take a look?

Also about yoru comment:

There are a couple of code changes in MIGRATION_3_4.

^ this should have never happened... if it did it's a problem on our part... Can you please pinpoint the commit for us?

the only fix i can think of is for you to add a patch (in fhircore) that changes the migration step 4 to 5, in order to "catch up" with the migration steps released in fhir engine... and going forward that shouldn't be a problem.

jingtang10 commented 1 year ago

more important - @ellykits we really should start thinking about getting fhircore to the released version of engine. @pld @FikriMilano @brynrhodes we should evaluate the level of effort for this one - this could be really beneficial for us in the long run.

ellykits commented 1 year ago

@jingtang10 @aditya-07. After further investigations with @ndegwamartin, we figured the issue to be on our released version of the engine library. In most cases, our releases are ahead of the SDK. Sometimes we merge in unmerged SDK commits as we await the FHIR SDK release. In this particular scenario, we published and used the library with some code changes in the MIGRATION_3_4 migration and missed out on the updates that were later added.

jingtang10 commented 1 year ago

Thanks @ellykits... I guess in this case my proposed fix still applies. You might just need to change the migration step 4 to 5 to do what you missed out in migration step 3 to 4... does this make sense?

This does highlight the need for us to bring fhircore to released versions of engine. The "contract" will be clearer that way and we will be able to better support. Do we think this is achievable imminently or are there still gaps to address?

@pld

aditya-07 commented 1 year ago

@ellykits You should be able to mitigate the issue by adding the missing parts from MIGRATION_3_4 to your MIGRATION_4_5 and updating the app.

ellykits commented 1 year ago

@

Thanks @ellykits... I guess in this case my proposed fix still applies. You might just need to change the migration step 4 to 5 to do what you missed out in migration step 3 to 4... does this make sense?

Yes, that does. This will resolve our current issue.

pld commented 1 year ago

Thanks @ellykits... I guess in this case my proposed fix still applies. You might just need to change the migration step 4 to 5 to do what you missed out in migration step 3 to 4... does this make sense?

This does highlight the need for us to bring fhircore to released versions of engine. The "contract" will be clearer that way and we will be able to better support. Do we think this is achievable imminently or are there still gaps to address?

@pld

That's definitely what we want. The ongoing challenge is when we have a hard deadline (driven by partner timelines related to the movement of people) and we need to use code that isn't yet in a versioned release. This is trending in the correct direction, towards on using released versions as Android FHIR SDK and OpenSRP 2 become more mature

ellykits commented 1 year ago

I have managed to successfully apply the latest migrations. The issue can be closed at your convenience @jingtang10, if Peter's comment above is satisfactory.

jingtang10 commented 1 year ago

Thanks @ellykits... I guess in this case my proposed fix still applies. You might just need to change the migration step 4 to 5 to do what you missed out in migration step 3 to 4... does this make sense?

This does highlight the need for us to bring fhircore to released versions of engine. The "contract" will be clearer that way and we will be able to better support. Do we think this is achievable imminently or are there still gaps to address?

@pld

That's definitely what we want. The ongoing challenge is when we have a hard deadline (driven by partner timelines related to the movement of people) and we need to use code that isn't yet in a versioned release. This is trending in the correct direction, towards on using released versions as Android FHIR SDK and OpenSRP 2 become more mature

Thanks Peter. Can we draw up a list of issues blocking this shift? If you give me some pointers I can start a tracker.

ndegwamartin commented 1 year ago

@jingtang10 our custom artifact releases contain the following unmerged PRs cc @pld @ellykits