Update how we get referrals in Flow 2: Primero-> Progres

daissatou2 commented 3 years ago

Background, context, and business value

Update Flow 2: Primero-> Progres in how we get referral data from Primero (Flow 2-1)

Currently in job Flow 2-1, on a scheduled basis OpenFn...

Step 1: Gets cases from Primero that have been recently updated. See filtering criteria. This returns JSON response that includes data for cases and related services.
Step 2: And then filters for Services that have been recently updated.

However, we learned that the service date is not updated when the "REFER" button is clicked on the frontend. So there is no way to get the most recent referrals with this current implementation.

The specific request, in as few words as possible

See data flow diagram: https://lucid.app/lucidchart/c12010a3-c6a9-4169-bd02-16a3b7d29e57/edit?invitationId=inv_665431b4-228d-45bc-88e5-3b4779964ea1&page=jK.jmqob8Sa0#

We will need to replace step 2 above with new steps that will...

Get referrals for the cases returned in the step 1 getCases(..) request. GET /api/v2/cases/:id/referrals where id is each case record id returned in step 1. This will return JSON like this.
Find new referrals: Filter for referrals where created_at_date is greater than or equal to the datetime of the last job run (or the manual cursor).
For each new referral, use the service_record_id (see here) to then determine which service(s) from the step 1 cases response we should map & send to DTP.

1 Service = 1 Referral... so this whole process will help us to determine how many and which services to send to Progres.

state.json

Primero Gambella open_function

adaptor

Primero adaptor Primero API docs

expression.js

https://github.com/OpenFn/primero-progres/blob/master/jobs/2.a.getCases.js

For testing

The job will basically determine... Which Primero cases have been recently updated (getCases)? Which of these cases have new referrals (getReferrals for a case)? If new referrals found, which services to send to Progres (using service_record_id)?

aleksa-krolls commented 3 years ago

@lakhassane @daissatou2 sent an invite for later to review if you think it would be helpful to talk this one through... it's kind of a lot of steps just to figure out which services to send :)

lakhassane commented 3 years ago

@aleksa-krolls @daissatou2 a draft of the complete job has been provided. We added steps to getReferrals before sending to inbox.

I am not able to test yet considering getReferrals is hitting a 500. Also updated the job in prod to use primero@v2.3.5

aleksa-krolls commented 3 years ago

@lakhassane let's talk about this/ work on this together Monday, bc you shouldn't be hitting a 500... GET /api/v2/cases/:recordid/referrals

aleksa-krolls commented 2 years ago

@lakhassane Job 2a fails with TypeError: https://www.openfn.org/projects/p5am6e/runs/06176afd-38be-7621-94e7-f0913323baf5

Let @daissatou2 ready for a re-test.

lakhassane commented 2 years ago

@daissatou2 this was a typo in one of the line. It's corrected and here is the last run: https://www.openfn.org/projects/p5am6e/runs/06176ca0-270c-7ada-be6c-b05e2bf32002

daissatou2 commented 2 years ago

Seeing this issue now: https://www.openfn.org/projects/p5am6e/runs/06176e4f-c15b-7e76-beed-dee328647067 @lakhassane

daissatou2 commented 2 years ago

@lakhassane thanks for resolving the adapter issue above. I think the job might be fetching duplicate data...when I rerun it, it posts the same referrals that I fetched already earlier--instead of saying there is nothing more to post.

See here: https://www.openfn.org/projects/p5am6e/runs/06176e74-1ec4-73b0-a8f9-91a2ac00491d https://www.openfn.org/projects/p5am6e/runs/06176e82-e973-78b3-8ee8-216b80517830

aleksa-krolls commented 2 years ago

@lakhassane in addition to the datetime cursor issue @daissatou2 flagged above, I'm also finding that we're sending ALL services for a case that has 1 referral (rather than only that service that was referred with matching service.unique_id and referral.service_record_id).

See this case where id: 9021d216-d8b3-4cde-a33c-ad3ed033cd68 https://demo-cpims-gambella.primero.org/v2/cases/9021d216-d8b3-4cde-a33c-ad3ed033cd68

This case only has 1 referral for 1 alternative_care service, yet 2 services are being sent to DTP - see run failure: https://www.openfn.org/projects/p5am6e/runs/06176fc6-c88c-7fac-a034-55981599e628

The second service is not complete and was not referred... and therefore should not be sent to DTP. Let me know if helpful to discuss.

daissatou2 commented 2 years ago

@lakhassane please use 686e5c7e-9f67-4de4-acf0-822e4c1a1c30 for testing today. It has one referral from today and a few from before today.

aleksa-krolls commented 2 years ago

@lakhassane how's this going? lmk if anything I or @daissatou2 can help with today like making new test data

lakhassane commented 2 years ago

@daissatou2 the cursor issue is corrected. However I might have a question.

I though that we send to DTP services where service_record_id (of fetch referrals) are among the unique_ids of services_section of a case? cc @aleksa-krolls

aleksa-krolls commented 2 years ago

@lakhassane As discussed, the job is timing out when posting to DTP: https://www.openfn.org/projects/p5am6e/runs/06179765-6f96-763d-b8ef-71de700fc82b

Please test locally to check if our request is hanging or you encounter 500 which might indicate DTP is down. I want to 100% confirm the issue is not on our side. cc @daissatou2

lakhassane commented 2 years ago

@aleksa-krolls I think it was a DTP issue. The job is running fine locally and now also in prod: https://www.openfn.org/projects/p5am6e/runs/0617aa65-2286-75d3-8e7e-f559c710dfbc

aleksa-krolls commented 2 years ago

@daissatou2 Confirming this is looking good? Can we close this issue?

OpenFn / primero-progres