search

OpenFn / unicef-cambodia

UNICEF Cambodia - Primero Interoperability
https://openfn.github.io/unicef-cambodia/
1 stars 2 forks source link

Add paging to `f1-j1-getPrimeroCasesV2.js` #130

Open aleksa-krolls opened 3 months ago

aleksa-krolls commented 3 months ago

Request

Primero API requests are now limited to a maximum of 1,000 records. While it's rare that we an API query returns more than 1,000 records, we should proactively update our OpenFn jobs to consider this limit and to handle paging if more than 1k records do exist.

In the job V2/f1-j1-getPrimeroCasesV2.js, we need to (1) change the parameter per to one thousand per: 1000 and then (2) handle scenarios where multiple pages of data may be returned

expression.js

Please branch off main and update the job V2/f1-j1-getPrimeroCasesV2.js . MAKE SURE TO ONLY UPDATE JOB IN V2 FOLDER: https://github.com/OpenFn/unicef-cambodia/blob/master/jobs/V2/f1-j1-getPrimeroCasesV2.js

Query to update: https://github.com/OpenFn/unicef-cambodia/blob/57237db68676e1eb1d6acf73fd36e506a227c391/jobs/V2/f1-j1-getPrimeroCasesV2.js#L19-L29

state

Primero (to generate state):

{
"configuration": "Primero V2 Cambodia Demo API user",
"data": {}
}

To test

Feel free to run locally. When ready to test, pls merge to the staging branch so that we can test on the Staging platform project: https://www.openfn.org/projects/primero-oscar-cambodia-io-staging/jobs/jywaga

Toggl

UNICEF Cambodia Primero IO Support [2024]

mtuchi commented 3 months ago

@aleksa-krolls should i branch off from main or staging ?, Because the project link on v1 uses staging branch and not main

aleksa-krolls commented 3 months ago

hey @mtuchi pls branch off main... and we can overwrite staging with your changes. (I don't trust what's currently on staging... so we should "refresh" that branch.)

mtuchi commented 3 months ago

Hiya @Aleksa Krolls there is a potential performance implication when using getCases with the { withReferrals: true } What this does is fetch referrals for each retrieved case which can take a while if there is a lot of cases fetched.

I am currently experimenting pagination with a small batch, I will share more tomorrow

aleksa-krolls commented 3 months ago

@mtuchi feel free to query in smaller batches (less than 1000)... we have long-running jobs enabled on platform

mtuchi commented 3 months ago

@josephjclark This is ready for code review now, See #131

mtuchi commented 3 months ago

@aleksa-krolls i have tested the changes locally with the lastRunDateTIme: 2024-06-20T07:12:25.383Z. Although i am getting cases from primero but when we filter to get oscarRefs i am getting empty results. Let me know if you need to sync to test this.

FYI. The changes in #131 uses @openfn/language-primero@2.11.7

aleksa-krolls commented 3 months ago

hey @mtuchi fyi I've asked @daissatou2 for help testing here - and then I will do a final review/merge to prod when I get back online

daissatou2 commented 2 months ago

@mtuchi I want to test this on the staging project but it looks like the PR was created against main. Can you please create the PR against staging?

mtuchi commented 2 months ago

@daissatou2 i think we need to overwrite what is on staging right now because i did ask @aleksa-krolls which branch to use and she suggested we use main

daissatou2 commented 2 months ago

@aleksa-krolls we tested this together locally and it passed. I created a case & referral on Primero and we ran the workflow. We did not merge the PR per Mtuchi's last comment.

aleksa-krolls commented 2 months ago

@mtuchi as discussed, pls refresh staging branch based on the main branch... and then we can merge your PR to staging

@aleksa-krolls i have refreshed staging branch and change the branch base from master to stagin in #131