[Quest/eCBIS] - Missing QuestionnaireResponse in the DB

[FikriMilano]

Describe the bug There are some QuestionnaireResponse that exist in the server but not in the app DB. It happens with family-registration response. This cause inconsistency, for the family Group resource to exist in the DB, but not the paired QuestionnaireResponse.

To Reproduce Steps to reproduce the behavior:

1. Run the app using ecbis-saa appId
2. Login with theecbis user
3. Do and finish the initial sync
4. Open App Inspection in the Android Studio
5. Run this query SELECT * FROM ResourceEntity WHERE resourceId="4b907e81-6067-45cd-813f-370c35748bc6"

Expected behavior It supposed to exist.

Screenshots

Smartphone (please complete the following information):

• Device: Pixel 3 emulator
• Android version: Android 12
• Build number: [e.g. Settings -> About phone -> Build number]
• Version: API 32

Additional context It exist in the server https://fhir.labs.smartregister.org/fhir/QuestionnaireResponse/4b907e81-6067-45cd-813f-370c35748bc6

[FikriMilano]

This is related to sync, will start looking from there

[FikriMilano]

It turns out, the app is syncing QuestionnaireResponse based on subject.organization This works fine if the main extracted resource is Patient. The response got synced properly. Because Patient resource does have that search parameter https://build.fhir.org/patient.html#search

But in this case where the main extracted resource is Group. It won't work, because Group resource doesn't have subject.organization search parameter. Instead it's using subject.managing-entity https://build.fhir.org/group.html#search

[FikriMilano]

Wait, no. This QuestionnaireResponse is not synced even when using subject.managing-entity=400 as well. But it does have that managingEntity value. https://fhir.labs.smartregister.org/fhir/QuestionnaireResponse/4b907e81-6067-45cd-813f-370c35748bc6

This feels like a HAPI FHIR server bug @dubdabasoduba

[FikriMilano]

@f-odhiambo The work around is to sync QuestionnaireResponse without the organization filtering. Let me know if this is ok, for now.

[pld]

@f-odhiambo The work around is to sync QuestionnaireResponse without the organization filtering. Let me know if this is ok, for now.

Would this also be the longer term fix?

[FikriMilano]

Wait, no. This QuestionnaireResponse is not synced even when using subject.managing-entity=400 as well. But it does have that managingEntity value. https://fhir.labs.smartregister.org/fhir/QuestionnaireResponse/4b907e81-6067-45cd-813f-370c35748bc6

This feels like a HAPI FHIR server bug @dubdabasoduba

No, that's the quick fix. As for short term or long term, it depends on when this server side bug managed to be fixed.

If it's managed to be fixed then I can simply use advanced filtering that have both search parameter, for better sync performance.

[pld]

Cool, is there an open bug in the HAPI tracker for this issue or does this have to do with our extensions?

On Jul 26, 2022, at 18:38, Fikri Milano @.***> wrote:

 Wait, no. This QuestionnaireResponse is not synced even when using subject.managing-entity=400 as well. But it does have that managingEntity value. https://fhir.labs.smartregister.org/fhir/QuestionnaireResponse/4b907e81-6067-45cd-813f-370c35748bc6

This feels like a HAPI FHIR server bug @dubdabasoduba

No, that's the quick fix. As for short term or long term, it depends on when this server side bug managed to be fixed.

If it's managed to be fixed then I can simply use advanced filter that have both search parameter, for better sync performance.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[FikriMilano]

I am not sure if it's the extension, but I can reach out to @dubdabasoduba

I was thinking to confirm the bug to the server team before creating a HAPI tracker.

So, can we agree to use the workaround for now?

[pld]

This issue isn't urgent until we're going to start a pilot, which we have a couple months until (I think @AnnieMungai and @Gental-Giant would know better), so for know I'm marking this blocked, you can let the server team take a look and see what they think, if there's nothing resolving this before we get closer to the date of a pilot we can go ahead w/a quick fix

[FikriMilano]

Got it, that's fine

[dubdabasoduba]

@FikriMilano Do we know why the QR sync is based on these attributes? It doesn't seem right to tie a resource sync to its references.

[FikriMilano]

What do you mean by QR?

[dubdabasoduba]

@FikriMilano sorry QuestionnaireResponse

[FikriMilano]

Oh I see.

So in the past, we have a goal to improve the sync duration performance. Then these attributes that I mention helps in achieving that goal, because it limits to only sync QuestionnaireReponses that related to that organization

[FikriMilano]

@dubdabasoduba Do you have any updates from the server side?

[dubdabasoduba]

@FikriMilano I think the fix then is with how we sync the data. i.e how we define the sync parameter and then use them for sync.

My assumption here is we are taking about sync down from the server. Just to make sure I am answering the correct things

[FikriMilano]

Thanks Ben. Honestly I don't see anything wrong on the way we define the sync parameters for this case (let me know if there is though).

You can have a look and try to replicate the HAPI server bug w this. https://fhir.labs.smartregister.org/fhir/QuestionnaireResponse?_count=300&_filter=subject.organization%20eq%20400%20or%20subject.managing-entity%20eq%20400

Then try to search using cmd+f then type 4b907e81-6067-45cd-813f-370c35748bc6

[dubdabasoduba]

@FikriMilano the problem is around how the resources are defined. Not all resources will have attributes we can tag locations, organisations or careteams to. This means we might need to use extensions for those that do have.

Also, the attributes that allow the tagging of locations, organisations or careteams don't have the same names across the resources. This means writing a generic sync configuration isn't very easy.

[dubdabasoduba]

4b907e81-6067-45cd-813f-370c35748bc6

So based on this it means the sync to the server is the problem right? If so then we need to make sure sync to the server is not restricted based on any attributes. All the device created data should be synced to the server without any restrictions.

[FikriMilano]

So based on this it means the sync to the server is the problem right? If so then we need to make sure sync to the server is not restricted based on any attributes. All the device created data should be synced to the server without any restrictions.

Yeah, that might actually the case

[dubdabasoduba]

@dubdabasoduba I think I will advise keeping the blocked tag on this. This should be fixed after this discussion is complete https://github.com/opensrp/fhircore/discussions/746

[FikriMilano]

cc @pld @f-odhiambo

[f-odhiambo]

@FikriMilano based on our discussions I think we can approach the closure of this ticket in this manner

1. Identify the problemeatic groups that have these issues
2. We can implement an error handling mechanism in the event that we need to edit the group resource (family)
3. Notify the user that there is a missing QuestionnaireResponse, perhaps we can look at how do we repopulate and create a new QR
4. Close current issue with points 1-2 and then raise issues in point 3 as an enhancement ticket

Feel free to add or comment on any missed discussed items

[FikriMilano]

Additional discussed items:

1.1 Remove the problematic Group resources

5. Sync Group resources that only has the QR. If the QR is missing, then don't sync the QR (not sure how we able to do this though)

[pld]

following up on this, can we remove blocked now that work on sync strategies has begun, or at least be more specific and refer to the ticket that this is dependent on?

[dubdabasoduba]

Technical implementation specific

• [ ] Sync all the data found on the Local SQLite DB to the FHIR server when sync is automatically or manually triggered.
• [ ] Confirm that QRs are correctly tagged based on this issue.

NB: - This issue does not need to handle sync down to the server since the Remote Sync issues defined here will adress that.

[FikriMilano]

Let's have this points moved to a new issue