[Remote Sync] Fetching a lot of data referenced on the composition resources throws an SQLiteQuery: exception

[dubdabasoduba]

To Reproduce Steps to reproduce the behaviour:

1. Use the Composition resource and sync_config.json on this branch
2. Add it to the app/debug or stage server then try to load the app.

Expected behaviour

• [ ] All the resources referenced on the Composition should be synced correctly

Additional context

E/SQLiteQuery: exception: Row too big to fit into CursorWindow requiredPos=100, totalRows=101; query: SELECT *
            FROM LocalChangeEntity
            ORDER BY LocalChangeEntity.id ASC
E/FhirEngineImpl$syncDownload$$inlined$collect: Save remote and resolved conflicting resources to database failed
    android.database.sqlite.SQLiteBlobTooBigException: Row too big to fit into CursorWindow requiredPos=100, totalRows=101
        at android.database.sqlite.SQLiteConnection.nativeExecuteForCursorWindow(Native Method)
        at android.database.sqlite.SQLiteConnection.executeForCursorWindow(SQLiteConnection.java:1001)
        at android.database.sqlite.SQLiteSession.executeForCursorWindow(SQLiteSession.java:838)
        at android.database.sqlite.SQLiteQuery.fillWindow(SQLiteQuery.java:62)
        at android.database.sqlite.SQLiteCursor.fillWindow(SQLiteCursor.java:161)
        at android.database.sqlite.SQLiteCursor.onMove(SQLiteCursor.java:131)
        at android.database.AbstractCursor.moveToPosition(AbstractCursor.java:248)
        at android.database.AbstractCursor.moveToNext(AbstractCursor.java:280)
        at com.google.android.fhir.db.impl.dao.LocalChangeDao_Impl$9.call(LocalChangeDao_Impl.java:310)
        at com.google.android.fhir.db.impl.dao.LocalChangeDao_Impl$9.call(LocalChangeDao_Impl.java:297)
        at androidx.room.CoroutinesRoom$Companion.execute(CoroutinesRoom.kt:78)
        at androidx.room.CoroutinesRoom.execute(Unknown Source:7)
        at com.google.android.fhir.db.impl.dao.LocalChangeDao_Impl.getAllLocalChanges(LocalChangeDao_Impl.java:297)
        at com.google.android.fhir.db.impl.DatabaseImpl$getAllLocalChanges$2.invokeSuspend(DatabaseImpl.kt:205)
        at com.google.android.fhir.db.impl.DatabaseImpl$getAllLocalChanges$2.invoke(Unknown Source:8)
        at com.google.android.fhir.db.impl.DatabaseImpl$getAllLocalChanges$2.invoke(Unknown Source:2)
        at androidx.room.RoomDatabaseKt$withTransaction$2.invokeSuspend(RoomDatabase.kt:58)
        at androidx.room.RoomDatabaseKt$withTransaction$2.invoke(Unknown Source:8)
        at androidx.room.RoomDatabaseKt$withTransaction$2.invoke(Unknown Source:4)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89)
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:169)
        at kotlinx.coroutines.BuildersKt.withContext(Unknown Source:1)
        at androidx.room.RoomDatabaseKt.withTransaction(RoomDatabase.kt:51)
        at com.google.android.fhir.db.impl.DatabaseImpl.getAllLocalChanges(DatabaseImpl.kt:204)
        at com.google.android.fhir.impl.FhirEngineImpl.getConflictingResourceIds(FhirEngineImpl.kt:146)
        at com.google.android.fhir.impl.FhirEngineImpl.access$getConflictingResourceIds(FhirEngineImpl.kt:44)
        at com.google.android.fhir.impl.FhirEngineImpl$syncDownload$3$1.invokeSuspend(FhirEngineImpl.kt:101)
        at com.google.android.fhir.impl.FhirEngineImpl$syncDownload$3$1.invoke(Unknown Source:8)
        at com.google.android.fhir.impl.FhirEngineImpl$syncDownload$3$1.invoke(Unknown Source:2)
        at androidx.room.RoomDatabaseKt$withTransaction$2.invokeSuspend(RoomDatabase.kt:58)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source:1)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source:1)
        at androidx.room.RoomDatabaseKt$acquireTransactionThread$2$2.run(RoomDatabase.kt:121)
        at androidx.room.TransactionExecutor$1.run(TransactionExecutor.java:47)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923)

NB: - Do not these configs to the eCBIS preview server since this is being used by clients to test the app.

[pld]

try increasing the cursor window? https://stackoverflow.com/questions/51959944/sqliteblobtoobigexception-row-too-big-to-fit-into-cursorwindow-requiredpos-0-t/66215674#66215674

[dubdabasoduba]

try increasing the cursor window? https://stackoverflow.com/questions/51959944/sqliteblobtoobigexception-row-too-big-to-fit-into-cursorwindow-requiredpos-0-t/66215674#66215674

This makes sense but we have to set the max size we want for the window. In case we set a high number that the device can't handle the app might end up being closed by the OS

[dubdabasoduba]

We got this because of the large measures data set we are fetching all at once when we move the measure references to the Composition Resource.

I think we can update how we query the resources referenced on the Composition resources. We could probably batch them i.e if we have 19 Measures we need to fetch them in batches of 4 or so instead of all at once.

[dubdabasoduba]

So probably combine the 2 options to make sure we don't increase the cursor window too much but also account for a situation the size might be exceeded.

[f-odhiambo]

This may have been fixed by adding the resources to the Composition resource and tested on the current implementation on the gateway CC @dubdabasoduba