MSF testing feedback for workflow refactor

aleksa-krolls commented 2 weeks ago

Issue

I added back console.logs so we can actually see the events payload you're sending to the dhis2 api... here is the latest payload causing errors.

See run failure with these errors:

{
  "status": "ERROR",
  "validationReport": {
    "errorReports": [
      {
        "message": "Event: `YKPaC6XbCtE`, Enrollment value is null.",
        "errorCode": "E1033",
        "trackerType": "EVENT",
        "uid": "YKPaC6XbCtE"
      },
      {
        "message": "Event: `oI0JAlb7eEv`, Enrollment value is null.",
        "errorCode": "E1033",
        "trackerType": "EVENT",
        "uid": "oI0JAlb7eEv"
      },
      {
        "message": "Event: `ScWv14dZJCe`, Enrollment value is null.",
        "errorCode": "E1033",
        "trackerType": "EVENT",
        "uid": "ScWv14dZJCe"
      },
      {
        "message": "Event: `kI6rFF9cSBF`, Enrollment value is null.",
        "errorCode": "E1033",
        "trackerType": "EVENT",
        "uid": "kI6rFF9cSBF"
      },
      {
        "message": "Event: `I9ogwuEDiP5`, Enrollment value is null.",
        "errorCode": "E1033",
        "trackerType": "EVENT",
        "uid": "I9ogwuEDiP5"
      },
      {
        "message": "Event: `NlGWQCzjq0l`, Enrollment value is null.",
        "errorCode": "E1033",
        "trackerType": "EVENT",
        "uid": "NlGWQCzjq0l"
      },
      {
        "message": "Event: `FSXFyEC4FlE`, Enrollment value is null.",
        "errorCode": "E1033",
        "trackerType": "EVENT",
        "uid": "FSXFyEC4FlE"
      },
      {
        "message": "Event: `GIEwOFq6S5U`, Enrollment value is null.",
        "errorCode": "E1033",
        "trackerType": "EVENT",
        "uid": "GIEwOFq6S5U"
      },
      {
        "message": "Event: `fvElC0XZTFc`, Enrollment value is null.",
        "errorCode": "E1033",
        "trackerType": "EVENT",
        "uid": "fvElC0XZTFc"
      },
      {
        "message": "DataElement `iEdamWl0wzQ` is not valid: `value_not_zero_or_positive_integer`",
        "errorCode": "E1302",
        "trackerType": "EVENT",
        "uid": "hDMY0TQezaD"
      },
      {
        "message": "Value cgi-s_score|0 is not a valid option for iEdamWl0wzQ DataElement in option set 1,2,3,4,5,6,7",
        "errorCode": "E1125",
        "trackerType": "EVENT",
        "uid": "hDMY0TQezaD"
      },
      {
        "message": "DataElement `c4TiPRUQ56A` is not valid: `value_not_zero_or_positive_integer`",
        "errorCode": "E1302",
        "trackerType": "EVENT",
        "uid": "hDMY0TQezaD"
      },
      {
        "message": "Value cgi-s_score|0 is not a valid option for c4TiPRUQ56A DataElement in option set 1,2,3,4,5,6,7",
        "errorCode": "E1125",
        "trackerType": "EVENT",
        "uid": "hDMY0TQezaD"
      },
      {
        "message": "Value 2 is not a valid option for tsFOVnlc6lz DataElement in option set 0_4,5_9,10_14,15_19,>20",
        "errorCode": "E1125",
        "trackerType": "EVENT",
        "uid": "hDMY0TQezaD"
      }
    ]

My Errors Analysis

There are 4 issues here:

For the error Enrollment value is null, only 1 of these events is linked to a TEI or enrollment (see L222). I think this means something is wrong in the Upsert TEI step or the mappings. **The other events are not linked to a trackedEntity at all!
- See my comment on #34** with guidance on how to solve this.
- We expect each TEI to have 1 enrollment in the program w9MSPn5oSqp, and if not - we enroll them in step wf2/2-upsert-teis.js.
- Each enrollment has an id, which we should include in this events mapping like this:
```
{
"program": "w9MSPn5oSqp",
"orgUnit": "OPjuJMZFLop",
"trackedEntity": "gVukOXy6qks",
"enrollment": "BCVoZgbAFLu", //should include and get from earlier step
"occurredAt": "2024-11-08T16:52:49.000",
"programStage": "xrCTheIzyDV",
"dataValues": [ ... ] 
}
```
Error "message": "Value 2 is not a valid option for tsFOVnlc6lz DataElement in option set 0_4,5_9,10_14,15_19,>20",
- On L252 we're sending the value 2 instead of 0_4... why? We should send 0_4 as per the getRangePhq custom transformation.
For the errors related to Data Element c4TiPRUQ56A
- On L240 looks like we were sending cgi-s_score|0.
- There was an issue with the optionSet mapping that I'm flagging to the client.
- In the meantime, I've updated the optionSet sheet... can you please regenerate the collection and lmk if this resolves the issue or if it's still outstanding?
For the errors related to Data Element iEdamWl0wzQ
- On L244 looks like we're sending cgi-s_score|0
- In the meantime, I've updated the optionSet sheet... can you please regenerate the collection and lmk if this resolves the issue or if it's still outstanding?

mtuchi commented 2 weeks ago

@aleksa-krolls any update here ?

daissatou2 commented 2 weeks ago

@mtuchi the Create Events step is failing, see run. I was attempting to sync a new MSF Mental Health - mhGAP Closure submission for the patient with MSF ID: IQ146-24-000-742.

aleksa-krolls commented 1 week ago

hey @mtuchi I've updated this issue description with my issues of the errors thrown in recent run failures. Please address these and then hand back to Aicha for testing.

@daissatou2 please see this video for guidance on how to troubleshoot and record any other errors like these going forward. Thanks!

mtuchi commented 1 week ago

EOD Update

Added a condition that filter patient without trackedEntity and enrollment and logged the patient id
Logged the encountersMapping payload on create events step.
Convert matching option with the value of no to FALSE and yes to TRUE

Test the latest changes here

aleksa-krolls commented 1 week ago

@mtuchi good news is that I was able to resolve most of the event errors! Will walk you through it tomorrow.

But now the wf is failing on next-staging when trying to update the TEI's gender in the last step. See run failure and error.

Failed step Update TEIs after 202ms
{
  "message": "TypeError: Cannot destructure property 'trackedEntity' of 'TEIs[answer.person.uuid]' as it is undefined.",
  "name": "RuntimeError",
  "severity": "fail",
  "source": "runtime",
  "subtype": "TypeError"
}

mtuchi commented 1 week ago

Hey @aleksa-krolls i have filtered person without trackedEntity in update-teis step and add a log of the person uuid. The changes are already synced to app you can proceed with testing

OpenFn / openfn-lime-pilot

MSF testing feedback for workflow refactor #61

Issue

My Errors Analysis