WF1 Modifications for the Iraq Pilot

aleksa-krolls commented 1 week ago

Background, context, and business value

Read the README from the original prototype implementation to refresh your memory on Workflow 1. Also see this video for a walk through.

The specific request, in as few words as possible

For the Iraq pilot implementation, we want to modify the wf1 jobs to update and extend the mappings.

See diagram workflow - TECHNICAL for updated specs.

wf1-1-getTEIs (dhis2)

Then see this diagram for updated query parameters:

program: ‘w9MSPn5oSqp’ , 
orgUnit: ‘OPjuJMZFLop’, 
programStatus: 'ACTIVE'

wf1-2-getLocations (dhis2) --> new step

Add a new workflow step to GET /api/optionGroups/kdef7pUey9f?fields=id,displayName,options[id,displayName,code] in order to return a list of place of living locations, that you will need to map OMRS address in the next step. The response will look like this payload.

We'll use this^ response to dynamically generate a mapping table (that might look like something like this if we wanted to hardcode this table).

wf1-2-createPatients (omrs) --> rename to wf1-3

Before you map Patient data, you will need to send a POST request to 'idgen/identifiersource/8549f706-7e85-4c1d-9424-217d50a2988b/identifier' to generate a unique identifier for each payload. (This step should be the same as the prototype.)

Then we should update and add field mappings according to this WF1 mapping spec.

see rows 8-9 in mapping spec for how to map the location options from the prior step

See payload for an example request body that looks similar to what you need to send to OpenMRS when you create Patients (POST '/patient').

wf1-3-updateTEIs (dhis2) --> rename to wf1-4

See diagram or below sample code for how the mappings in this step should be updated.

program: w9MSPn5oSqp, 
orgUnit: OPjuJMZFLop, 
attributes: [
  {attribute: "P4wdYGkldeG", value: ${patient_number}, //DHIS2 patient number to use as lookup key
   {attribute: "AYbfTPYMNJH", value: ${patient.uuid}, //OMRS patient uuid   
   {attribute: "ZBoxuExmxcZ", value: ${patient.identifier[0].identifier}} //id generated in wf1-2 e.g., "IQ146-24-000-027"
]

API Documentation

DHIS2 latest docs: https://docs.dhis2.org/en/develop/using-the-api/dhis-core-version-239/metadata.html
OMRS REST API docs: https://rest.openmrs.org/#openmrs-rest-api

Input

Credentials

DHIS2: See LP MSF DHIS2 UAT - June 2024
OpenMRS: See LP MSF OpenMRS Demo

Data

Run wf-1-1 to get TEIs from DHIS2

Output

If DHIS2 returns X patients, then X number of patients should be created in OpenMRS. See example output payload: https://github.com/OpenFn/openfn-lime-pilot/blob/main/sample-data/pilot/openmrs_postPatientBody.json

Toggl

MSF 2024

aleksa-krolls commented 1 week ago

@mtuchi this is the first MSF issue ready for devpt today - more to come tomorrow. Ping me on Slack if any questions or if helpful to talk through this.

aleksa-krolls commented 1 week ago

@mtuchi I added links to API documentation above. For querying /optionGroups from dhis2, fyi this is leveraging the Metadata API, which doesn't have a ton of docs specific to optionGroups, but generally describes how to interact with and filter this API. But fyi I have tested the http GET request I want to send on postman to verify it returns the data we need. https://docs.dhis2.org/en/develop/using-the-api/dhis-core-version-239/metadata.html?h=optiongroup+develop+2.39#webapi_browsing_the_web_api

mtuchi commented 1 week ago

I will pick this up on Monday

aleksa-krolls commented 6 days ago

hey @mtuchi as discussed

I updated the mapping spec with the FULL json paths (e.g., person.gender)
See this payload for an example output to send to OMRS

mtuchi commented 5 days ago

EOD Update

I have manage to update the mapping for create patients but two fields are not mapped yey(commented out) [countryDistrict and cityVillage]. I have also add mapping for locations, statusMap and nationalityMap in the get-locations step.

@aleksa-krolls you can proceed with testing on https://app.openfn.org/projects/cd92dd57-9a3c-4318-bdcb-f57a386cf811/w/94e04fb5-d5ae-45ad-be31-98b902f36861?v=9

There are couple of improvements on adaptors that i have i noticed when i was working on this issue, i will open a PR on adaptors for those changes

aleksa-krolls commented 2 days ago

hey @mtuchi some WF1 feedback... please *create a new branch off staging** to see what I was testing with.

Something is off with the filtering by cursor in WF1. See run where my input is { "manualCursor": "2024-09-13T08:00:00.00" } and yet # of TEIs to migrate to OMRS (after filtering) is ZERO! I expect at least the TEI eWXRNHtmHB0 to be returned because its lastUpdated is AFTER my cursor (2024-09-13T11:17:48.386 > 2024-09-13T08:00:00.00).
```
{
"created": "2019-06-13T10:43:47.558",
"orgUnit": "OPjuJMZFLop",
"trackedEntityInstance": "eWXRNHtmHB0",
"trackedEntityType": "cHlzCA2MuEF",
 "lastUpdated": "2024-09-13T11:17:48.386",
```
I'm noticing the workflow executes all steps, even if NO teis found to sync to OMRS. Can you pls update the edge conditions accordingly? Feel free to test and implement on app with this input and then sync to GH, if easier.

mtuchi commented 2 days ago

I have update the teis to use updatedAt instead of lastUpdatedAt which doesn't exist. I have also add edge condition

OpenFn / openfn-lime-pilot