opensrp / fhircore

FHIR Core / OpenSRP 2 is a Kotlin application for delivering offline-capable, mobile-first healthcare project implementations from local community to national and international scale using FHIR and WHO Smart Guidelines on Android.
https://smartregister.org
Apache License 2.0
50 stars 39 forks source link

Review $apply and StructureMaps with Logical Models implementation on the SDK #2697

Open dubdabasoduba opened 10 months ago

dubdabasoduba commented 10 months ago

Describe the feature request.

Implementation plan

Implementation plan (For Engineers) The plan for implementing the solution e.g. via a description or a check list for the various ordered tasks that will need to be completed. i.e. Describe how you intend to solve the problem

owais-vd commented 10 months ago

Based on the discussion with @maimoonak I've created a supply data as bundle of resources that contains the IMMZDT08.json library with all the dependents libraries and Plan-Def and Activity-Def. For example: the bundle looks like

{
    "resourceType": "Bundle",
    "id": "IMMZ.DT.08.Measles-bundle",
    "type": "transaction",
    "entry": [
        {
            "resource": {
                "resourceType": "PlanDefinition"
            }
        },
        {
            "resource": {
                "resourceType": "ActivityDefinition"
            }
        },
        {
            "resource": {
                "resourceType": "Library"
            }
        }
    ]
}

When i run the test then get this exception: Could not resolve expression reference 'Provision of the MCV dose' in library 'IMMZDT08'. Although this definition is available on IMMZDT08.cql file.

Tried to convert the IMMZDT08.cql file using fhircore-tooling then get another exception during conversion.

java.lang.IllegalStateException: failure 'Could not compile CQL File. Errors:
8:1-8:26: Could not resolve model info provider for model FHIR, version 4.0.1.

Also tried to convert the IMMZDT08.cql file using CqlBuilder class but still getting the following exception.

Could not compile CQL File. Errors:
10:1-10:33: Could not load source for library IMMZCommon, version null.
11:1-11:33: Could not load source for library IMMZConcepts, version null.
12:1-12:33: Could not load source for library IMMZConfig, version null.
13:1-13:40: Could not load source for library IMMZVaccineLibrary, version null.
14:1-14:28: Could not load source for library FHIRCommon, version null.
97:3-97:41: Could not resolve library name IMMZCom.
185:8-185:42: Could not validate reference to expression MCV Doses Administered to Patient because its definition contains errors.
122:11-122:52: Could not resolve library name IMMZCon.
171:5-171:31: Could not validate reference to expression High Transmission Setting because its definition contains errors.
178:14-178:53: Could not validate reference to expression First MCV Dose Administered to Patient because its definition contains errors.
28:2-28:42: Could not validate reference to expression Second MCV Dose Administered to Patient because its definition contains errors.
144:14-144:23: Could not resolve library name IMMZCom.
200:8-200:42: Could not validate reference to expression MCV Doses Administered to Patient because its definition contains errors.
105:14-105:48: Could not validate reference to expression MCV Doses Administered to Patient because its definition contains errors.
131:2-131:39: Could not validate reference to expression No MCV Doses Administered to Patient because its definition contains errors.
35:7-35:28: Could not validate reference to expression Needs Birth MCV Dose because its definition contains errors.
48:5-48:26: Could not validate reference to expression Needs Birth MCV Dose because its definition contains errors.
159:7-159:41: Could not validate reference to expression MCV Doses Administered to Patient because its definition contains errors.
64:5-64:26: Could not validate reference to expression Needs Birth MCV Dose because its definition contains errors.
70:7-70:28: Could not validate reference to expression Needs Birth MCV Dose because its definition contains errors.
82:7-82:54: Could not validate reference to expression Supplimentary MCV Dose Administered to Patient because its definition contains errors.
206:14-206:61: Could not validate reference to expression Supplimentary MCV Dose Administered to Patient because its definition contains errors.
90:2-90:28: Could not validate reference to expression Provision of the MCV dose because its definition contains errors.
112:20-112:54: Could not validate reference to expression MCV Doses Administered to Patient because its definition contains errors.
118:9-118:43: Could not validate reference to expression MCV Doses Administered to Patient because its definition contains errors.
165:14-165:53: Could not validate reference to expression Birth MCV Dose Administered to Patient because its definition contains errors.
193:14-193:54: Could not validate reference to expression Second MCV Dose Administered to Patient because its definition contains errors.

cc: @pld @f-odhiambo @dubdabasoduba

pld commented 9 months ago

This is the branch w/our testing, https://github.com/google/android-fhir/compare/master...opensrp:android-fhir:test-who-cql-lib

pld commented 9 months ago

I'm going to close this, we've reviewed and concluded that it does not run.

On performance, we are seeing positive signs on the execution speed of the z-score content, this is much more complex than that but still this is a good sign

dubdabasoduba commented 9 months ago

We also reviewed the Google examples an generated the following resources

Patient resource

{"resourceType":"Patient","id":"7c5d71ae-26b7-4ba4-9fac-f8359c35f390","name":[{"family":"Woman","given":["Test"]}],"gender":"female","birthDate":"1995-09-13"}

CarePlan Before filling a task

{"resourceType":"CarePlan","id":"40cdaa71-b0b7-471e-8c53-ce1a0ac881ea","instantiatesCanonical":["http://localhost/PlanDefinition/PlanDefinitionAdultPatient"],"status":"active","description":"CarePlan of Record","subject":{"reference":"Patient/7c5d71ae-26b7-4ba4-9fac-f8359c35f390"},"activity":[{"outcomeReference":[{"reference":"Encounter/145b757f-d77f-41ae-9bc9-b4a2467b219b"}],"reference":{"reference":"Task/3c8f9828-7c6d-4aad-9fc5-3db7989d47c8"},"detail":{"status":"completed"}},{"reference":{"reference":"Task/bd3ea109-4b72-425f-9722-c097ff87cc85"},"detail":{"status":"not-started"}},{"reference":{"reference":"Task/f8509c21-d180-4063-ba43-e1b0940604ca"},"detail":{"status":"not-started"}},{"reference":{"reference":"Task/2b12f1ed-ad1a-44b3-8fa7-cf2f7186723d"},"detail":{"status":"not-started"}}]}

Diabetes Task

{"resourceType":"Task","id":"3c8f9828-7c6d-4aad-9fc5-3db7989d47c8","extension":[{"url":"http://hl7.org/fhir/aphl/StructureDefinition/condition","valueExpression":{"description":"Adult Patient","language":"text/cql.identifier","expression":"Eligible for Diabetes Mellitus Screening"}}],"identifier":[{"system":"http://ncd.org","value":"diabetes-mellitus-screening-task"}],"basedOn":[{"reference":"CarePlan/40cdaa71-b0b7-471e-8c53-ce1a0ac881ea","type":"CarePlan"}],"status":"completed","intent":"order","code":{"coding":[{"system":"http://snomed.info/sct","code":"171183004","display":"Diabetes mellitus screening"}]},"description":"Diabetes Mellitus Screening","focus":{"reference":"Questionnaire/DiabetesMellitusScreening"},"for":{"reference":"Patient/7c5d71ae-26b7-4ba4-9fac-f8359c35f390"},"executionPeriod":{"end":"2023-09-13T11:06:31+03:00"},"lastModified":"2023-09-13T11:00:54+03:00","owner":{"reference":"PractitionerRole/81417c6a-a6ab-453c-a26b-c08791161045","display":"Community Health Worker"},"restriction":{"period":{"end":"2024-09-12T11:00:54+03:00"}}}

Observation

{"resourceType":"Observation","id":"29548b74-0338-40c8-895a-51450e82d4df","code":{"coding":[{"system":"http://snomed.info/sct","code":"271061004","display":"Random blood glucose measurement"}],"text":"Random blood glucose measurement"},"subject":{"reference":"Patient/7c5d71ae-26b7-4ba4-9fac-f8359c35f390"},"encounter":{"reference":"Encounter/145b757f-d77f-41ae-9bc9-b4a2467b219b"},"valueInteger":40}

Condition

{"resourceType":"Condition","id":"4f980de5-ffec-4c4b-b21d-194c55092622","code":{"coding":[{"system":"http://snomed.info/sct","code":"161915001","display":"No general symptom"}],"text":"No general symptom"},"subject":{"reference":"Patient/7c5d71ae-26b7-4ba4-9fac-f8359c35f390"},"encounter":{"reference":"Encounter/145b757f-d77f-41ae-9bc9-b4a2467b219b"}}

Encounter

{"resourceType":"Encounter","id":"145b757f-d77f-41ae-9bc9-b4a2467b219b","subject":{"reference":"Patient/7c5d71ae-26b7-4ba4-9fac-f8359c35f390"}}

CarePlan After Task filling

{"resourceType":"CarePlan","id":"e0727022-f1fe-4950-9543-d81d40eb9232","instantiatesCanonical":["http://localhost/PlanDefinition/PlanDefinitionAdultPatient"],"status":"active","description":"CarePlan of Record","subject":{"reference":"Patient/7c5d71ae-26b7-4ba4-9fac-f8359c35f390"}}
dubdabasoduba commented 9 months ago

The error we are getting when we try extract the QR to the our resources

Exception executing transform create('http://fhir.org/guides/who/smart-immunization/StructureDefinition/IMMZ-C-register-client') as model on Rule "QRtoPatient": Unknown Resource or Type Name 'http://fhir.org/guides/who/smart-immunization/StructureDefinition/IMMZ-C-register-client'
org.hl7.fhir.exceptions.FHIRException: Exception executing transform create('http://fhir.org/guides/who/smart-immunization/StructureDefinition/IMMZ-C-register-client') as model on Rule "QRtoPatient": Unknown Resource or Type Name 'http://fhir.org/guides/who/smart-immunization/StructureDefinition/IMMZ-C-register-client'
jingtang10 commented 9 months ago

Discussed with @dubdabasoduba, @DebbieArita and @owais-vd:

We think the transformation support service in HAPI actually does load structure definitions from an IG: https://github.com/hapifhir/org.hl7.fhir.core/blob/7d4816f62832661c36809cf846a7bd043f63d22e/org.hl7.fhir.validation/src/main/java/org/hl7/fhir/validation/TransformSupportServices.java#L38

And we think this is why the transformation works in matchbox according to Luke. AI here: chat with Oliver to figure out how in matchbox they are feeding the package to the tranformation support services and/or the simpler worker context.

fhircore uses its own implementation of transformation support services, and if the resource is undefined it does not look through structure definitions but simply calling out to the resource factory, which only handles known resources: https://github.com/opensrp/fhircore/blob/da81d9d4d923e841d00f8f64328fe75a0a1b8ff9/android/engine/src/main/java/org/smartregister/fhircore/engine/util/helper/TransformSupportServices.kt#L82

so we need to change the code here too.

pld commented 9 months ago

Is there an Android FHIR SDK API for TransformationSupport we should be using?

On Sep 26, 2023, at 19:06, Jing Tang @.***> wrote:

 Discussed with @dubdabasoduba, @DebbieArita and @owais-vd:

We think the transformation support service in HAPI actually does load structure definitions from an IG: https://github.com/hapifhir/org.hl7.fhir.core/blob/7d4816f62832661c36809cf846a7bd043f63d22e/org.hl7.fhir.validation/src/main/java/org/hl7/fhir/validation/TransformSupportServices.java#L38

And we think this is why the transformation works in matchbox according to Luke. AI here: chat with Oliver to figure out how in matchbox they are feeding the package to the tranformation support services and/or the simpler worker context.

fhircore uses its own implementation of transformation support services, and if the resource is undefined it does not look through structure definitions but simply calling out to the resource factory, which only handles known resources: https://github.com/opensrp/fhircore/blob/da81d9d4d923e841d00f8f64328fe75a0a1b8ff9/android/engine/src/main/java/org/smartregister/fhircore/engine/util/helper/TransformSupportServices.kt#L82

so we need to change the code here too.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.

dubdabasoduba commented 9 months ago

There is one on HAPI https://github.com/hapifhir/org.hl7.fhir.core/blob/7d4816f62832661c36809cf846a7bd043f63d22e/org.hl7.fhir.validation/src/main/java/org/hl7/fhir/validation/TransformSupportServices.java#L15. I think we should be extending this (if need be) or using it as is.

jingtang10 commented 9 months ago

Really godo point - I do think the knowledge manager library might be able to provide an API that passes structure maps to the extraction process @pld. But the highest priority now is it work out how to do this, we can upstream the functionality after this.

dubdabasoduba commented 9 months ago

@pld @jingtang10 I think we added this because the HAPI FHIR class uses R5 Context which FHIR Core isn't implementing yet.

joiskash commented 9 months ago

I have created a fork with a hacky solution https://github.com/joiskash/fhircore Please check the main branch and run the same test "perform extraction for patient registration"

  1. I copied TransformSupportServices and ConceptMapEngine from the matchbox repo. Removed anything to do with r5. I don't know if this really helped.
  2. Trimmed down both the structure maps QRToLM and LMToPatient to just transform uniqueId.
  3. Created a custom object by extending Base and called it IMMZ_LM. I'm not sure if the name / resource type really matters.
  4. Used that as the target for the first transform with SM QRToLM
  5. Then used that as a source and Patient as a target with SM LMToPatient.

Instead of loading the StructureDefinition in the context, I just created the object and passed that. I'm sure this is not the ideal way, but it works. Its very messy, but before I put any more effort into it, I wanted to know if we are happy with this and can move forward for the time being.

joiskash commented 9 months ago

I have updated the code to now load the ig as an NPM package in the SimpleWorkerContext. This way the TransformSupportServicesMatchBox class can find the structure definition and create the resource and type for the Logical Model. The StructureMapUtilities class uses the transform support service and creates the resource and type on the fly using the Structure Definition.

dubdabasoduba commented 9 months ago

@joiskash @jingtang10 What was the agreement on the next steps?

joiskash commented 9 months ago

@joiskash @jingtang10 What was the agreement on the next steps?

Changes made to make the SM work:

  1. Load the who.immunization implementation guide as a npm package into the SimpleWorkerContext instead of "hl7.fhir.r4.core". We need to check what the repercussions of this change is.
    val immunizationIg =
        "content/general/who-eir/packages/package.r4.tgz"
    val contextR4 =
        SimpleWorkerContext.fromPackage(
          NpmPackage.fromPackage(
            File(
              ClassLoader.getSystemResource(immunizationIg).file
            ).inputStream()),true).apply {
          setExpansionProfile(Parameters())
          isCanRunWithoutTerminology = true
        }
  2. Remove all [x] from value[x] in the structure maps. For some reason the transform throws an error that you cannot set the property valueString of answer. Using just answer.value instead of answer.valueString in the structure map solved this problem.
  3. Commented out
item.answer first as answer where item.linkId = 'administrativeArea' then {
      answer.value as coding -> immzc.administrativeArea as area then {
        coding -> area.coding = coding "set coding value";
        coding.display as display -> area.text = display "set display";
      } "set coding";
    } "first answer for administrativeArea";

coding is not a primitive type. You cannot set it. I guess this is a SM issue

  1. Commented out
    immzc.sex as sex -> patient.gender = translate(sex, 'http://fhir.org/guides/who/smart-immunization/ConceptMap/IMMZ.C.SexToAdministrativeGender', 'code') "set gender";

Unknown code DE7. Not sure why this happens. Need to dig deep.

  1. Copied https://github.com/ahdis/matchbox/blob/7fa5c7794b89f1f87c87b288f52ffb39068146d7/matchbox-engine/src/main/java/ch/ahdis/matchbox/mappinglanguage/TransformSupportServices.java#L33 class. replaced r5 by r4.
  2. Copied https://github.com/ahdis/matchbox/blob/7fa5c7794b89f1f87c87b288f52ffb39068146d7/matchbox-engine/src/main/java/ch/ahdis/matchbox/mappinglanguage/ConceptMapEngine.java class. Replaced r5 by r4. I could not find an equivalent to import org.hl7.fhir.r5.model.Enumerations.ConceptMapRelationship; so I just removed the condition isOkRelationship(t.getRelationship()) and the isOkRelationship function.
  3. Passed an instance of the TransformSupportServicesMatchbox to StructureMapUtilities.
delcroip commented 9 months ago

Regarding the coding issue: here an example https://github.com/WorldHealthOrganization/smart-emcare/blob/3b0410a1dd8a173ceb19844ae9331a6e6cc28af7/input/mapping/emcare.b10-16.signs.2m.p.map#L813

concept.coding = create('Coding') as coding,
                coding.system = 'https://fhir.dk.swisstph-mis.ch/matchbox/fhir/CodeSystem/emcare-custom-codes',
                coding.code = 'EmCare.B10S2.DE04'