OpenFn / openfn-lime-pilot

MSF LIME Project - OpenFn Workflows for Iraq Pilot
1 stars 1 forks source link

Modify WF1 and WF2 to use the `metadata_mapping.json` file on github #52

Closed AishaHassen closed 2 weeks ago

AishaHassen commented 1 month ago

Background, context, and business value

The current implementation of WF1 and WF2 use hardcoded metadata mapping in the workflows. The ask on this issue is to replace the hardcoded metadata mappings with a metadata.json file that is stored on github.

The specific request, in as few words as possible

  1. Add a step to fetch the metadata.json file
  2. use that for the rest of the workflow that require the use of a metadata mapping

TODO: Spec better once I understand the structure of the JSON file

Expected data volumes

How many records do we think these jobs will need to process in each run? For example:

When you GET data from Primero, this may return up to 1000 records. There are no known Primero API limits for # of records, but there is API paging to consider.

state.json

Either provide state directly, or link to a file. If sensitive information should be in state, redact it and provide instructions for where it can be found.

{
  "configuration": ["SEE LAST PASS: 'client cred'"],
  "data": { LINK TO STATE },
  "cursor": "2020-01-19 00:00:00"
}

expression.js

For new jobs, describe the number of jobs needed and the high-level function of each job. Also include the trigger on platform and the adaptor needed for each job. For existing jobs, provide a link to the job itself in Github and the high-level changes needed to be made. _Provide the information below for each job that is required._

job name

adaptor

trigger

operation

output

Toggl

Name of the Toggle project to log work

mtuchi commented 1 month ago

In #55 I am fetching the metadata file from GitHub and map all existing variables with the data from the metadata file. One thing to note is we don't have mapping for encounter-date in all of the mapping. I think this should be added in the excel file. For example to get yUT7HyjWurN: 'encounter-date', for mhgapFollowup. In the excel sheet F04-mhGAP Follow-up we should have External ID: encounter-date and DHIS2 DE UID : yUT7HyjWurN