Update F1-J1 to getReferrals for each case and only map referred services to Oscar

[daissatou2]

Feature Request

Background, context, and business value

The f1-j1-getPrimeroCasesV2 job currently maps cases and services which have NOT been referred to Oscar. Instead, the job should only be syncing services which have been referred to Oscar.

The specific request, in as few words as possible

The job currently has two getCases operations. We are only making changes after the first getCases operation.

The job should work like the getCases job in the Primero-Progres integration where it getReferrals for each case returned from the first getCases operation, and then only maps referred services to Oscar. For each case returned in the first getCases operation, the job should

1. getReferrals. See line 45 in Primero-Progres job for example.
2. Filter to only map referred services to Oscar (AK to advise here).

state.json

{
"configuration": "Primero V2 Cambodia Demo (openfn_cp_user)",
}

adaptor

language-primero

expression.js

ONLY EDIT THE JOB IN THE V2 FOLDER. f1-j1-getPrimeroCasesV2

[aleksa-krolls]

@taylordowns2000 Given the breaking change in Primero v2.7.0, I think we should refactor this job as follows:

1. In the first getCases(...) step, now enable the option withReferrals: true to eliminate the getReferrals(...) step on L31 and combine operations.

2. Then, in the second getCases(...) step, we should apply a filter to NOT return any cases where case.services_section[*] contains ANY service where service_response_type: 'referral_to_oscar' because this logic is being handled in the 1st get. See my comments and suggested code: https://github.com/OpenFn/unicef-cambodia/blob/master/jobs/V2/f1-j1-getPrimeroCasesV2.js#L66-L71

@daissatou2 can you please create some test data for Taylor on Monday and work with him to implement?

[daissatou2]

@taylordowns2000 please test with:

1. "case_id": "9485b839-ef42-400e-9fe2-7f1d5c2e4714" should NOT be returned in our getCases() because it was not referred
2. "case_id": "19673a21-31e6-4053-ac7d-04dee24bda6f" should be returned in the first getCases()

Note: I'm unsure about the purpose of the second getCases if we are filtering out service_response_type: 'referral_to_oscar'. Isn't the purpose of this job to only get cases that were referred to Oscar?

[taylordowns2000]

@daissatou2 , this is ready for testing - please see the additional logs. Note that in my testing I wasn't able to find "other cases" before or after filtering. Could there be something amiss with the query we pass to primero for that second getCases(...)?

[daissatou2]

@taylordowns2000 seeing this error: https://openfn.org/projects/primero-oscar-cambodia-io-staging/runs/062545e4-993c-7501-b677-cceac10b2ac4

[taylordowns2000]

@daissatou2 , job needed to use v2.7.0 . I've updated it and it's running: https://openfn.org/projects/primero-oscar-cambodia-io-staging/runs/062545fc-181b-7905-9ec3-6a0892324a67

[daissatou2]

@taylordowns2000 @aleksa-krolls I am seeing this new error and I'm not sure if it's related to the changes made on this issue but was not seeing it before these changes.

I have verified that the db-id for the referral status field and values is correct in the job.

https://openfn.org/projects/primero-oscar-cambodia-io-staging/runs/06254607-df0f-77eb-9da9-85bc4889338c

[daissatou2]

@taylordowns2000 The old error was because my test data didn't have a referral actually, so our job is behaving well. I will create a separate issue so that if no case with services is found in F1-J1, F2-J2 outputs a log that says "no data to upsert" instead of failing.

[daissatou2]

Seeing an issue where 3 of the 4 cases I created in Primero are not being synced to Oscar. Will do some analysis/troubleshooting in the morning. I'll reach out if I need help @taylordowns2000

https://openfn.org/projects/primero-oscar-cambodia-io-staging/runs/06255b77-0a4a-74df-94c2-1fda36037609