daissatou2 commented 2 years ago

Feature Request

Background, context, and business value

We are rewriting the f2-j2-upsertCasesToPrimero job which upserts case and referral data fetched from Oscar to Primero.

The specific request, in as few words as possible

See the data flow diagram. LucidChart login is in LP.

The f2-j2-upsertCasesToPrimero should have two main functions: 1)send Oscar cases (with or without services) to Primero 2) send Oscar decisions on Primero referrals to Primero. Follow the steps below for each function of the job.

Process 1: Sending Oscar cases to Primero

Given an array of Oscar cases that match the criteria below, upsert cases in Primero using oscar_number or case_id as the UUID.

If oscar.resource != 'Primero' || oscar.resource===undefined ||(oscar.resource == "Primero" && oscar.status=== "Referred"), then this is an Oscar case we need to sync.
Log cases that meet this criteria as "Cases without decisions ".
ThenupsertCase() using the mapping specifications. In column A we have marked the fields that should be sent in this step and you can filter by this column. -If oscar.external_id is undefined, upsert the Primero case and related services using "oscar_number": "oscar.global_id" as the external_id upsert key. -If oscar.external_id is defined, upsert Primero case and related services using "case_id": "oscar.external_id" as the external_id upsert key.

*We assume 1 oscarService = 1 primeroService. For each record in `oscar.services[],  we will upsert 1 Primerocase.services_section`.  SEE MAPPING SPECS FOR SERVICE MAPPING RULES.**

Process 2: Sending Oscar decisions to Primero

Given an array of Oscar cases that match the criteria below, upsert cases AND referrals in Primero usingcase_id as the UUID.

If oscar.resource = "primero"  && oscar.status!= "Referred" this is a decision on a Primero referral.
Log cases that meet this criteria as "Oscar decisions".

Referral-level update For each matching case ...

Get referrals: GET /cases/:id/referrals using the oscar.external_id. See sample JSON response.
If no referrals are found, log an error.

For each Oscar service  where  oscar.services.enrollment_date = null loop through these referrals until we find the matching referrals where service_record_id = "oscar.services.uuid"

Once we find the matching referral records to update, pass the ids onto the next operation to update each referral...
updateReferral(...) to PATCH an update to the Primero status field with what is in the referral_status field from the Oscar response. PATCH /cases/:record_id/referrals/:id See sample JSON payload for request body

Mappings notes for referral level update: if we get value "Accepted" from Oscar status field, send to Primero status: 'accepted' if we get value "Exited" from Oscar status field, send to Primero status: 'rejected'

Case-level update: upsertCase() using the mapping specifications.

In column B we have marked the fields that should be sent in this step. You can filter by this column.
For the upsert external_id key, use case_id: oscar.external_id
Service to update:services_section[*].unique_id: o.services.uuid
Referral status field: services_section[*]. referral_status_ed6f91f =  accepted_850187 || rejected__74769 based on oscar.status

state.json

Primero: Oscar (to generate state):

{ "configuration": "Oscar Staging (api only user)", "data": "Data will be generated from this job" }

{ "configuration": "Primero V2 Cambodia Demo (openfn_cp_user)", "data": "Run F2-J1 to generate state" }

adaptor

language-primero

expression.js

To re-write this job... consider creating a new file to preserve the original expression. [f2-j2-upsertCasesToPrimero](https://github.com/OpenFn/unicef-cambodia/blob/master/jobs/f2-j2-upsertCasesToPrimero.js)

aleksa-krolls commented 2 years ago

@grayey @taylordowns2000 Here is the long-awaited spec for rewriting the UNICEF job. Let @daissatou2 know if you want to jump on a call to walk through this today.

grayey commented 2 years ago

I estimated wrongly. This appears to be more than a 2 . Should be a 3-4. I'll update as a 4 to err on caution. (Using 5 as 4 is absent as a poker point). The next operation on the job is to begin upserting since the maps are done.

@taylordowns2000 cc: @aleksa-krolls @daissatou2

grayey commented 2 years ago

@daissatou2 any idea about this error when trying to upsert case? Complaining about certain fields.

I'm going away now. Still facing errors with upsertCase so I'm unable to test getCase. If this step is done, the next operation to updateReferrals is last.

taylordowns2000 commented 2 years ago

@grayey estimates that once we get a response on the blocking question there is about 1 hour of work remaining. He was unable to work on the updateReferrals section (a step when we send decisions) because he’s missing some information from Aicha.

taylordowns2000 commented 2 years ago

@aleksa-krolls , it looks like the only commits that were missed in Emeka's branch were the two that came after b1726ca.

Both were made on the 22nd by Aleksa, and it looks like they'll be pretty easy to reimplement.

If you give me a thumbs up, I'll get the current version from Emeka's branch onto master and you can lay down these two commits again. Does that work?

aleksa-krolls commented 2 years ago

@taylordowns2000 thumbs up! I'll then re-add my commits today

aleksa-krolls commented 2 years ago

@taylordowns2000 Some updates...

I re-integrated relevant old commits, as well as copied over multiple data transformation functions from the old job version. It doesn't look like Emeka had gotten to many of these data transformation steps yet for several mappings... will have Aicha review, but with my changes, I think we should be looking pretty good on the field-level mappings for Process 1 (see diagram).
I noticed many case-level mappings were incorrectly implemented on the service-level (and vice versa). My fixes here should resolve the Primero errors Emeka flagged to @daissatou2 last week.

Given that I copied over some data transformation functions from the old job, can you please review my commit to ensure everything has been implemented optimally? https://github.com/OpenFn/unicef-cambodia/commit/da49f939ecfb39a51f0fcae2e8bd59cf2612a19e

Then, we are far from being able to test. The job doesn't look close to being done.

Process 1 - This has not been implemented correctly. Two critical things I notice so far, but I think this needs review in entirety...
- There is no filtering applied to the cases being upserted in Primero. See my comment on L338.
- See Aicha's specified logic for which external_id to use in the upsert step.
Process 2 - It looks like this has not been implemented yet at all.

Keep us posted on how we can support this week.

taylordowns2000 commented 2 years ago

Operation 1 and 2 done, separated cases from decisions. Will pick up tomorrow AM.

taylordowns2000 commented 2 years ago

@aleksa-krolls , @daissatou2 - ready for you review. please see TODOs and last commit updating comments: https://github.com/OpenFn/unicef-cambodia/commit/fd699fec81cab711e1ee24d1cc34150d542c17bf

taylordowns2000 commented 2 years ago

@daissatou2 , all done! please update the job on prod to use v2.7.0 to access these new features

aleksa-krolls commented 2 years ago

@daissatou2 job version updated on platform: https://www.openfn.org/projects/primero-oscar-cambodia-io-staging/jobs/jv79xn On Monday, please test

daissatou2 commented 2 years ago

@taylordowns2000 the decision on referrals looks good.

However, in the services section, we are creating a new service instead of updating the existing one with the status. We should be updating the existing service (marked with refer again) with the Accepted status. Run link to sync decision back to Primero: https://openfn.org/projects/primero-oscar-cambodia-io-staging/runs/0625054e-2ad0-7bee-9e0a-59bd53349246 "external_id": "19673a21-31e6-4053-ac7d-04dee24bda6f",

taylordowns2000 commented 2 years ago

@daissatou2 , this is ready to test after https://github.com/OpenFn/unicef-cambodia/commit/e64c24f8eada08d52b94be0123ce14d7671a7113

daissatou2 commented 2 years ago

testing here is pending #107

See failed run: https://openfn.org/projects/primero-oscar-cambodia-io-staging/runs/06254481-b96e-7452-a663-c9528daaf58e

daissatou2 commented 2 years ago

Testing here is still pending #107, see latest comments there.

daissatou2 commented 2 years ago

Seeing this error: https://openfn.org/projects/primero-oscar-cambodia-io-staging/runs/062557f0-2f79-7a52-8cb0-10fa2cf0af1e

The run passes but there is a (node:189988) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'unique_id' of undefined error.

You can test locally with "case_id": "54b63bb9-1d70-41b6-9f52-3e7eadd1adbc", after running the f2-j1 job.

taylordowns2000 commented 2 years ago

@daissatou2 , this error happens when no matchingService is found. I've updated the logic, please confirm that this is desired: we don't update decisions if no matching service is found: https://github.com/OpenFn/unicef-cambodia/commit/7d3b148ead7c7618eac68ebc0d7e6d2c4a77009f

daissatou2 commented 2 years ago

@taylordowns2000 let's discuss. The job should have found a matching service for "case_id": "54b63bb9-1d70-41b6-9f52-3e7eadd1adbc

daissatou2 commented 2 years ago

@taylordowns2000 @aleksa-krolls I've tested sending one decision from Oscar to Primero and that passed. I set up 4 cases in Primero (that need to be synced to Oscar) so I can test syncing multiple decisions, and cases from Oscar to Primero in one run.

I'll need to troubleshoot an issue where 3 of the 4 cases I created in Primero are not being synced to Oscar. See comment under #107

OpenFn / unicef-cambodia

Rewrite the f2-j2-upsertCasesToPrimero job #104

Feature Request

Background, context, and business value

The specific request, in as few words as possible

Process 1: Sending Oscar cases to Primero

Process 2: Sending Oscar decisions to Primero

state.json

adaptor

expression.js