UNICEF Primero - UNHCR Progres - FLOW 2

[hnyende]

Background, context, and business value

UNICEF and UNHCR are building an interoperability layer between their two systems, Primero and Progres, in order to reduce inefficient, insecure, and duplicative data entry processes. OpenFn will sync the cases and referrals between these two systems.

Example: As a UNICEF caseworker, I would like to electronically send a referral for a case to UNHCR. I would also like to receive an update on the referral decision.

The specific request, in as few words as possible

• see interactions between Primero- OpenFn-DTP
  • See the data flow diagrams --> Check for FLOW 2
  • See the mapping sheet for field mappings and lookup tables.

Flow 2: Primero -> Progres

To start this flow, OpenFn will check Primero for referrals. This happens every hour.

Create Jobs 1 &2 in which OpenFn gets cases from Primero and gets the case and services, maps the Primero extract and upserts to Progres

1. Get referral data: GET /api/v2/cases -- where service_implementing_agency='UNHCR' This should return: https://github.com/OpenFn/primero-progres/blob/master/sampleData/primero_sample_state.json
2. Map Primero response to DTP/Progres interface. The field progres_primeroid will be the primary uid used by DTP.
3. Upload referrals to DTP/Progres: POST https://antirrio.azure-api.net/primero-uat/ReceiveIncomingReferral

Create Job 3 which will be triggered when a decision is received from DTP/Progres, and updates Primero case with decision status.

4. DTP sends decision to OpenFn: https://github.com/OpenFn/primero-progres/blob/master/sampleData/flow2_progresDecision.json
5. Upsert case in Primero with referral decisison: PATCH /api/v2/cases/:id ... we will need to first find the matching Primero case using progres_individualid

state.json Primero - See LP for Primero Dadaab Progres/DTP - Use the certificate

adaptor Primero adaptor DTP: HTTP?

Trigger Job 1: Scheduled every day Job 2: Flow job based on 1 Job 3 is triggered when a decision is received from DTP

expression.js 3 jobs

1. GET cases from Primero
2. Map Primero response to to DTP/Progres and upload referrals to DTP/Progres
3. On receiving DTP/Progres referral decision, upsert Primero with decision status

[aleksa-krolls]

@hnyende this issue looks really nice 👍 , but I think there's some confusion between the system names (Primero vs. DTP vs. Progres)... let's discuss in our check-in to then finalize this issue and the data flow diagram.

Also, this is is the destination endpoint for sending referrals. To discuss... https://antirrio.azure-api.net/primero-uat/ReceiveIncomingReferral

[aleksa-krolls]

@lakhassane Also ready to be worked on when you have finished up Flow 1... this is the other direction of the same business process. @hnyende is the lead on this issue, and can also support with questions :)

[aleksa-krolls]

@lakhassane When you're done drafting these, to enable easier testing... can you please setup job 1 to post the Primero cases back to the OpenFn inbox, and then configure job 2 to run on a Message filter trigger? (Before go-live we'll change back to flow jobs, but UNHCR wants to post their own JSON samples to the Inbox.)

@daissatou2 We can then share the JSON output generated by job 1 with Cyril and team... this will be the state for job 2 that they can then manually manipulate for their own tests.

[lakhassane]

@hnyende @aleksa-krolls some of the properties in the mapping sheet are inside arrays (e.g service_response_day_time) then they are present multiple times in the case we are mapping. Could you advise on the best way to handle those: which one to take? is there a combination to do ?

services_section": [
          {
            ...
            "service_response_day_time": "2021-04-19T11:53:49.000Z",
            "service_response_type": "service_provision",
            ...
          },
          {
            ...
            "service_response_day_time": "2021-04-22T16:23:11.000Z",
            "service_response_type": "service_provision",
            ...
          }
        ],

Also there are fields to map that are present multiple times (e.g progres_comments) is there a reason why?

[lakhassane]

@hnyende for job 3, can you advise on the mapping to use for the decision to post back to primero, the actual fields to send?

[aleksa-krolls]

@lakhassane Updates from our side...

1. If we fetch a case that has multiple service items in its services_section array, then we want to create 1 referral per service. (The case mappings like name, age, etc. will be the same for both referrals... but the service_type and related info will be different.)
2. When mapping user specific fields, Primero team says we need to make a separate call to the /users endpoint to fetch info on that user. We can filter the user search by owned_by to find a specific user. https://github.com/OpenFn/primero-progres/blob/5733a0bfb5d7548e83203b05bac87fc8920580f5/jobs/2.b.uploadReferrals.js#L20-L23

More feedback to come... thanks!

[aleksa-krolls]

@lakhassane as just discussed per the above feedback, let's adjust the job to...

1. First look-up user info via the /users API endpoint. (We only need to search for users specified in the case owned_by field.)
2. Then create a referral obj to send to DTP for every item in the services_section where service_implementing_agency === 'UNHCR
3. Then map as you are currently...

[aleksa-krolls]

@lakhassane please let me know when this done/ changes are committed... and of course, if you have any questions! We need to make minor mapping adjustments, so I want to make sure I'm working off the latest branch.

[lakhassane]

@aleksa-krolls the changes in here are done.

[aleksa-krolls]

@lakhassane please advise on what the token should look like for the updated configuration given your last changes.

[aleksa-krolls]

@lakhassane When we get the user details, it is returning undefined for this first referral where the case user is owned_by: primero. Can you please take a look? See run L7: https://www.openfn.org/projects/p5am6e/runs/rx8eddk4

(Wondering if it is an issue with how we're calling the /users endpoint or an access issue)