OpenFn / unicef-cambodia

UNICEF Cambodia - Primero Interoperability
https://openfn.github.io/unicef-cambodia/
1 stars 2 forks source link

Upsert cases in Oscar job times out #56

Closed aleksa-krolls closed 3 years ago

aleksa-krolls commented 3 years ago

Issue We receive 100+ cases when we fetch from the Primero API in f1-j1 (see run here).

This causes f1-j2-casesToOscar.js to time out - see run example 9988060.

To replicate Run f1-j1-getPrimeroCases.js. One question that might be affecting the data volume size here - are we filtering cases by dates properly when we fetch from Primero? Looking at the f1-j1 logs the date looks like it's in a weird format...

Found new cases, updating 'last created case' date.
Found cases w/ updates, updating 'last updated case' date.
The last transition update is now: 25-09-2020 11:26
The last transition creation is now: 25 11:26:28 +0000-09-2020
Finished.
taylordowns2000 commented 3 years ago

This no longer times out, as it's granted long_run permission. I don't think the f1-j1 cursor is working anymore, however, and there appears to be a load issue throwing a 422 in oscar.

aleksa-krolls commented 3 years ago

@taylordowns2000 Confirmed that the f1-j1 cursor is no longer working because it works fine when I hardcode a date value. This works okay for testing for now, but when you get the chance over the next day - can you please check this out?

Pretty sure the issue is with the transitions_created_at date format - see below snippet from the latest run. The last transition creation is now: 25 11:26:28 +0000-09-2020

taylordowns2000 commented 3 years ago

Looks like we've got a new format in the response from Primero. Checking it out now. image

taylordowns2000 commented 3 years ago

cursor fixed!