FHIR data from Big Querry to Superset dashboards

[AnnieMungai]

Is your feature request related to a problem? Please describe. With OpenSRP moving to a FHIR based product, the data model will change. The data will now be pulled from the HAPI server to the Big Query and from the Big Query to Superset.

Describe the solution you'd like

• [ ] Understand the FHIR data model.
• [ ] Look at the FHIR data model and figure out how to pull FHIR data from Big Querry, to Superset.
• [ ] Need to figure out how to write views on Big query that will be used to build the dashboards.

Could be linked to the completion of this issue: https://github.com/onaio/canopy/issues/2637

[alepietrobon]

thanks. i think we need to break this down because this is not a single ticket but more of an entire epic.

• understand the data model: this is dependent reading the FHIR docs but also being clear on what data is available right now in the server and what gets pulled from the connector. I suggest we start by identifying a set of indicators and have the OpenSRP team confirm that the data to calculate those indicators exists in the version of the app and the server that the beam connector reads from. @AnnieMungai , i wonder if this is a task for you or another PM, because i understand that we won't immediately be able to reproduce an existing dashboard. Once we know what we're trying to build for this V1 dashboard, then the Analytics team can pick it up and we can start navigating the data model and figuring out how to get those indicators reported reliably.
• Pulling from bigquery to akuko. This is an akuko ticket (https://github.com/onaio/akuko-app/issues/1302 ) that needs to be scheduled and followed up based on the akuko roadmap. It is a blocker for the analytics team to build dashboards, as doing the work on superset would probably be throw away
• The third part is the only ticket that is actionable by the analytics team. We can get started on that as we investigate ways to connect properly to the bigquery. We can figure that out as part of this resourcing request: https://onasystems.avaza.com/project/view/40252#!tab=task-pane&groupby=ProjectSection&view=vertical&task=722173&fileview=grid

[AnnieMungai]

@alepietrobon

• Based on the progress of the app right now, here are a few indicators that we are able to collect on the current app: they are mainly registrations. cc @dubdabasoduba

[muneneg]

@AnnieMungai I have been able to figure out part 3, the sample query can be found here

This query creates the table that can be used to calculated the sample indicators you had shared

For part 2 I think that is still work in progress.

For part 1, we are still learning, for now we used only 1 table.

cc @alepietrobon

[Gental-Giant]

For part 2 : Pulling from bigquery to akuko

• Matt confirms we can visualize on Superset

@AnnieMungai @alepietrobon @mberg

[pld]

I've updated this issue title and description to be about Superset, hopefully we can use the same or only slightly varied BigQuery queries in another other viz platform we use

[dubdabasoduba]

@muneneg For the representation of the indicators in this list here are the list of resources to fetch.

Household/family -- We are representing this on the Group Resource. Here is a sample of the Group.

How to structure the Query.

• Fetch a group with the following attribute
  • active = true
  • type = person
  • code = {"coding":[{"system":"https://www.snomed.org","code":"35359004","display":"Family"}]}

Household/Family members -- We are now referencing this on the member attribute on the Group resource.

This reflects the latest changes on how to represent a household on FHIR. Documented here

Please feel free to reach out for more information.

cc: @AnnieMungai @alepietrobon

[alepietrobon]

hi @dubdabasoduba , thanks for this. I remember the discussion on the family implementation as group, glad that's already done.

Just confirming, we don't expect any big changes to the data model for these indicators going forward, right?

[dubdabasoduba]

hi @dubdabasoduba , thanks for this. I remember the discussion on the family implementation as group, glad that's already done.

Just confirming, we don't expect any big changes to the data model for these indicators going forward, right?

Yes, I don't think we will be changing how households are represented anymore.

[pld]

I agree it's unlikely we'll change the family representation, we presented this to the WHO FHIR working group as it's implemented here, so we're reasonably committed.

One thing I expect we'll add, but it's not relevant yet is a way to query the Group resource that filters only those that represent families, we don't need that now because we don't use groups for anything else

On Apr 27, 2022, at 10:06, Benjamin Mwalimu @.***> wrote:

 hi @dubdabasoduba , thanks for this. I remember the discussion on the family implementation as group, glad that's already done.

Just confirming, we don't expect any big changes to the data model for these indicators going forward, right?

Yes, I don't think we will be changing how households are represented anymore.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[AnnieMungai]

@muneneg The family resources have been pushed to the BQ (see this issue). cc @dubdabasoduba @pld

• [ ] Task: Update the views/ dashboard with the new resources.

[pld]

I'm closing this issue b/c we now have Big Query data in Superset dashboards, we can open a more specific issue about these other tasks