OpenFn / openfn-lime

MSF's OpenFn Lime Prototype Project
1 stars 1 forks source link

Update wf2-3 job to use the same cursor as wf2-1 job #17

Closed daissatou2 closed 1 year ago

daissatou2 commented 1 year ago

Describe the bug and expected behavior

The cursor in workflow 2 job 3 is currently not linked to the cursor in job 1. This means that when I clear job state or update the cursor for job 1, the workflow properly fetches new/updated OMRS patients since the last sync (in job 1), however, it is not properly fetching the enrollments created since the last sync (in job 3).

Update the workflow to pass the cursor for job 1 to job 3. Job 3 should not be using it's own different cursor, it should be using the same cursor as job 1.

To Reproduce

In this run log, job 3 is using the cursor 2023-06-07T12:33:33.948Z (which is the time that the job ran). It should instead be using the same cursor from job 1 2023-06-01T07:50:00.000+0000 --see job 1's run log.

expression.js

Link to the job itself in Github:https://github.com/OpenFn/openfn-lime/blob/main/jobs/wf2-3-getEncounters.js
Adaptor: openmrs

state.json

Re-run job wf2-1 to generate state for wf2-3

DHIS2:

OpenMRS:

To test/resolve

  1. After the desired output is working locally (from the CLI), please [open a pull request].
  2. [Please test the change on OpenFn.org by re-running this run (link) and confirming success.]

Toggl

MSF LIME

mtuchi commented 1 year ago

@daissatou2 if the cursor is on job 1 does it mean the lastRunDateTime is suppose to be set on the last job ? job 5 or which job should set the lastRunDateTime ?

daissatou2 commented 1 year ago

If the cursor is set on job 1 shouldn't the lastRunDateTime also be set by job 1? Otherwise, we might miss some data that was created or updated in between when job 1 was triggered and job 5 finished running.

mtuchi commented 1 year ago

@daissatou2 This will be fixed in the latest improvements PR https://github.com/OpenFn/openfn-lime/pull/18