Update how f2-j2-upsertCasesToPrimero finds the correct Primero referral to update with Oscar decision

[daissatou2]

Feature Request

Background, context, and business value

For Flow 1 Primero -> Oscar, Oscar does not send the original Primero service.unique_id in their decision payload. This means we have to add additional steps and get creative to find the relevant Primero service which needs to be updated with the decision.

For example: Instead of something like this which includes the service.unique_id, we receive this.

The specific request, in as few words as possible

See the data flow diagram.

Step 1: After receiving a decision from Oscar, the V2 f2-j2-upsertCasesToPrimero job should follow the steps below to find the right referral to update.

1.For each case where resource = "Primero", Oscar will send a list of services in service[*]. Where service[*].enrollment_date = null, save the Oscar services[*].name in an oscarReferral array.

2. Then parse the decision JSON from Oscar to find the external_id. Use the external_id as case_id in getCases().
3. In this response, find the services where 1)service_response_type = referral_to_oscar and 2)referral_status_5fe9c1a is not defined OR does not equal neither 'accepted_270501' nor rejected_412652. This will return a list of services referred to oscar.
4. For every service name in oscarReferral, find the matching Primero service as follows mapping p.service_type: o.name.
   • Return the service where Primero services_section[]. service_type== the mapped version of serviceMap[oscarReferral] from the original oscar response. The mapping table is already in that job.
   • This should help us find 1 matching Primero service, which is the one we will update with the status.** If more than one service is returned with the same matching service_type, log an error.
5. For the matching service, save services_section[].unique_id as service_record_id. (You will use this variable again below).
6. Get referrals for the same case: GET /cases/:id/referrals using the case_id from step 2 above. See sample JSON response.
   • If no referrals are found, log an error.
   • If they are found, loop through these referrals until we find the referral where service_record_id = the service_record_id saved in step 5.
   • Once we find the matching referral record to update, pass the id onto the next operation... updateReferral(...) to PATCH an update to the Primero status field with what is in the referral_status field from the Oscar response. PATCH /cases/:record_id/referrals/:id See sample JSON payload for request body

Mappings notes for referral level update: if we get value "Accepted" from Oscar status field, send value "accepted" to Primero status field if we get value "Exited" from Oscar status field, send value "rejected" to Primero status field

Step 2: After updating the Primero referral, THEN also map the Oscar decision to the Primero service.

1. Use the service_record_id from step 5 above to update the service with the decision.

2. add a services_section[] mapping for Primero referral_status_5fe9c1a here.

   • Primero referral service_record_id = services_section[*].unique_id
   • The source field is the status field from the Oscar response. See mappings below.

Mappings notes for service level update: if we get status: Accepted from Oscar, send services_section[*].referral_status_5fe9c1a: 'accepted_270501' to Primero if we get status: Exited from Oscar, send services_section[*].referral_status_5fe9c1a: 'rejected_412652' to Primero

state.json

Primero:

{ "configuration": "Primero V2 Cambodia Demo API user", "data": "Run F2-J1 to generate state" } Oscar (to generate state):

{ "configuration": "Oscar Staging (api only user)", "data": "Data will be generated from this job" }

adaptor

Primero adaptor Primero API Docs

expression.js

f2-j2-upsertCasesToPrimero

Toggl

UNICEF Cambodia

[aleksa-krolls]

@lakhassane this one is ready to be worked on, but it's a little tricky. So let @daissatou2 and I know if you need to talk through it.

[aleksa-krolls]

@lakhassane Your estimate was 3 hours... how is this going? What's the status? Flagging that UNICEF may need to revoke our access today for the week as they work on data migration this week, so if you are unable to authorize... we now might need to wait until next week to finish and test if you didn't get this done yesterday.

[daissatou2]

Testing is pending Primero data migration.

[daissatou2]

Step 1 from above is working properly but Step 2 is not working

Currently, instead of updating the referral_status_5fe9c1a field on the existing service, the job is creating a new service and still NOT adding the status on that field either.

The job should be updating the referral_status_5fe9c1a field on the existing service using the steps outlined above for STEP 2.

[daissatou2]

Log for the issue: https://openfn.org/projects/primero-oscar-cambodia-io-staging/runs/0622896f-4c84-7d1e-835f-be80825e65aa

[daissatou2]

We did some investigating and here is what is currently happening: The f2-j2 job is using the same service_record_id to find the unqiue_id for all cases and referrals recieved from Oscar. See lines L297 and L321… service unique_id is the same: 3e3af14a-12f2-4717-9b43-442e102fc890 for two different cases. https://openfn.org/projects/primero-oscar-cambodia-io-staging/runs/0622a2ec-0810-76fb-b558-f9d65a0cbeb1

Is this service unique_id being written to 1 constant, rather than finding the service_record_id / unique_id for EACH relevant service on EACH case recieved from Oscar: https://github.com/OpenFn/unicef-cambodia/blob/master/jobs/V2/f2-j2-upsertCasesToPrimero.js#L261

Here's what should be happening: The job should be finding the service_record_id / unique_id for EACH relevant service on EACH case recieved from Oscar using the steps outlined above and in the the data flow diagram.

[daissatou2]

Hello @grayey how is this coming along? Any updates?

[daissatou2]

@grayey seeing the same error that was discussed in this comment.

In this run, we are upserting a decision for TWO services on TWO separate cases but we are using the same unique_id for both services...see lines 289 and 313.

[taylordowns2000]

@daissatou2 , this is ready for your testing. please note the TODO for you: https://github.com/OpenFn/unicef-cambodia/blob/master/jobs/V2/f2-j2-upsertCasesToPrimero.js#L251-L254

Once you're happy with how the tests are running, you can remove that TODO and restore the if/throw.

@grayey , your intuition was spot on but in the end later operations were referring to state.service_record_id - a string. In reality, there can be multiple cases at that point in the job and each will have its own unique service_record_id. (@aleksa-krolls and @daissatou2 , thanks for helping me see the r'ship between these.)

So.... as we go along we build a little Case:Service map. After the first loop where we update referrals I've added a log telling us what we've got for later use. Each key is a case_id and each value is a service_record_id:

Referrals updated. Case:Service ID map = {
  'cc803894-82b2-46c7-82fa-7c0ed50c9a9f': '4d770705-db07-4979-a46c-ee6cff572a65',
  'ed4c6461-1a7f-40b9-b857-cf22acddbe04': '07ac46e9-3725-4ef1-9167-127059cfff2e'
}

Finally, because of the way this job has been written, we've got to "pass along" the case_id to a bunch of different generator functions later in the job. Note my change to mapKeysToServices:l368 and classifyServices:l400

Hope this all makes sense. Holler if you want to chat through this later today. @grayey , It's still worth a refactor, IMO. Let's discuss how we can do that when you're back online at 3pm... assuming Aicha is happy with how the testing goes.

[daissatou2]

Thanks guys! The unique_id issue seems to be resolved.

However, according to the logs in this run, we are not sending the services_section[*].referral_status_5fe9c1a field to Primero. So the service status is actually not being updated in Primero.

See STEP 2 in the original issue above. Mappings notes for service level update: if we get status: Accepted from Oscar, send services_section[*].referral_status_5fe9c1a: 'accepted_270501' to Primero if we get status: Exited from Oscar, send services_section[*].referral_status_5fe9c1a: 'rejected_412652' to Primero

[taylordowns2000]

@daissatou2 , ready for testing! i've added 2 TODO blocks for @grayey ... there's some funky stuff going on with those mapKeysToServices and classifyServices functions.

[grayey]

ok @taylordowns2000. Taking a look now.

[aleksa-krolls]

Closing this issue. As discussed, @daissatou2 and I will prepare new specs for a new job version that will allow us to drop a lot of this legacy logic.