search

WorldHealthOrganization / smart-ig-starter-kit

Other
11 stars 12 forks source link

Build a Test Framework to evaluate L3 content such that it will run in L4 apps that use L4 common libraries #61

Open joiskash opened 7 months ago

joiskash commented 7 months ago

Questions:

  1. Should this run in an android emulator or cloud mobile devices? This tool should be designed to be platform agnostic. Since the SMART guidelines content should be able to run on other platforms/ devices too.
  2. Should this be an API ? If Android env does not have a specific implementation for extraction, cql evaluation etc. then this can be done.

Constraints:

  1. Support for FHIR R4 (?)
  2. Any more constraints ?

Objectives:

  1. To generate a compatibility matrix for various libraries in L4 Common tools that can execute a particular version of L3. a. Questionnaire Rendering b. FHIR Path evaluation c. CQL evaluation d. Data Extraction
joiskash commented 7 months ago

Bryn : There is a Postman collection with plan definition apply tests. There are 3 open source engines and 2 closed source that can be tested by this collection. Content from CPG, SMART Guidelines etc. have been used.

Jing: We should have similar tests in the SDK, this should be platform agnostic.

  1. Test whether L3 content runs vs Test if the L3 content was run, is the outcome correct?

We need to check if the guideline was followed in actual use. This testing framework could provide the methodology / reusable tools to evaluate an audit trail generated by an L4 application executing SMART Guidelines

@brynrhodes can you please share a link to the POSTMAN collection that you are talking about. We can have a look at it to understand what it does.

brynrhodes commented 6 months ago

https://github.com/cqframework/pd-apply/tree/main/connectathon/postman There are two folders in there, one for the ActivityDefinition, and one for the PlanDefinition tests

joiskash commented 6 months ago

Constraint:

  1. Support R4 and foresee eventual support for R5 such that there are no hard coded version constraints : Jose : Can we try to build it such that it does not depend on the version? Bryn: Its possible, but it has challenges performance, technical challenge etc. You can back port from R5 to R4. Jing: SDC, CQL support R4. We cannot switch to R5 with support from these IGs. Android SDK FHIR engine (search,storage, connect to fhir server etc.) support for R5 is a WIP. Jose: We do not need a real time support of multiple versions. Bryn: Depends on FHIR version that countries adopt. For new adoptions we can drive the version selection, but if there are other factors that result in a country implementing R5, then we need to have some answer.
costateixeira commented 5 months ago

I don't know how platform agnostic this could be, but I think that the L3 validation is meant for desktop platforms, because that is where the authoring itself works, right? concretely, I would like to add some of this validation to the pipelines (e.g. github).

joiskash commented 5 months ago

That makes sense.

  1. CQF engine

The android FHIR SDK internally uses org.opencds.cqf.fhir.cr.plandefinition.r4.PlanDefinitionProcessor (and other classes from org.opencds.cqf.fhir.cr.plandefinition.r4) to run $apply. So if we can maintain a FHIR server with org.opencds.cqf.fhir pointing to the same version as that used in the android fhir sdk then we are good to go. This could be part of android fhir sdks github pipeline. Any change in in the dependency could trigger creation of a docker image for a FHIR server with the right cqf dependency and the $apply api. The docker image can be run locally on authors machine / on a server. We can then write a script (not sure which language) which can run pointing to the SMART guideline IG to pick up content to test the PlanDefinitions, Measures etc. We need to think about the content here. Will the SMART guideline contain examples that can be used to test all the logic? What is the scope of all the tests? Happy scenario vs complicated use cases?

  1. SDC data extraction

We can take a similar approach as described above with org.hl7.fhir.r4.utils.StructureMapUtilities. This is assuming that all extraction logic will be written using StructureMaps. Definition based is slightly more difficult since this logic is written in the android fhir sdk.

joiskash commented 5 months ago

SOP mentions that examples scenarios / test results etc. are supposed to be packaged in the SMART guideline.

  1. Package common libraries in a deploy-able server. Docker image. Jose has worked on docker creation on IG. We do not need to trigger this in the android sdk github pipeline. We can always just pick fetch this/ change this.
  2. Test script language ideas - ExampleScenario, Requirements, TestPlan, TestScript FHIR test script etc.
  3. Large samples of Synthetic data wont be in the IG. One idea here is to keep a core IG and dependent IG (with all the examples and samples.
  4. We should check the stable version from the teams that are writing these library (Smile Cdr etc). There are other groups writing open source cql evaluators, this should run with that too.
  5. Where should this infra be maintained? Who is in charge of it? - WHO in charge of getting the IG to build / run @ritikarawlani please correct this if wrong.
  6. Test Matrix - Different evaluators, FHIR version, SDC Version

Initial scope can be small but we should make sure it is extendable. Call out the different type of tests that we do CQL evaluation, StructureMap extract etc. Need to complete this list.

joiskash commented 4 months ago

  1. Where will this docker image be hosted? WHO dockerhub? - revisit.

  2. Create a base image - we could start from hapi-fhir-jpaserver-starter - any other suggestions? Operations that need to be supported in the server:

    • $apply this maybe a separate docker image
    • $extract this maybe a separate docker image

  3. Create a script that can change pom.xml to include same dependencies used in a particular Android FHIR SDK release. If we find official docker images that use the same libraries as the android SDK for FHIR operations then we can just create a compose with those.

    • Jing : Automatically update FHIR dependencies in the SDK with Caveats (cc: @jingtang10 )

  4. Which SMART guideline is ready to start building this out? Need someone to help note down the exact test cases. Who will help with this?SMART Immunizations Measles

  5. List Test Framework -