search

OpenFn / wcs-wildmeat

Wildlife Conservation Society ConSocSci Project
1 stars 0 forks source link

Get Kobo Forms job: cursor not working #36

Closed aleksa-krolls closed 2 years ago

aleksa-krolls commented 2 years ago

Describe the bug

WCS has configured this job to run every 1 hour to get new submissions from Kobo. It should get submissions where end > state.lastRunTime (or since the manualCursor).

I think currently the same submissions are being fetched again and again. This is FLOODING the inbox, and also the job is timing out - see run: https://www.openfn.org/projects/wildmeat/runs/061a509e-1ebb-714d-9f4f-8451a276658b

To Reproduce

Only re-run the job with 1 form commented in so that you don't fill up the queue with hundreds of submissions... currently the job is fetch submissions for several Kobo forms.

expression.js

Please make sure this is working in both jobs:

  1. https://www.openfn.org/projects/wildmeat/jobs/jv9qxk https://github.com/OpenFn/wcs-wildmeat/blob/master/jobs/1.getKoboData.js
  2. https://www.openfn.org/projects/wildmeat/jobs/jvqq5b https://github.com/OpenFn/wcs-wildmeat/blob/master/jobs/fetch-rc-submissions.js

state.json

See LP for openfn_kobo

Configuration: 

{"openfnInboxUrl":"https://www.openfn.org/blah,"password":"pwd","type":"basic","username":"un"}

Expected behavior

Every run should only get NEW submissions where end has been updated since the last run (or manual cursor).

aleksa-krolls commented 2 years ago

@lakhassane As discussed... the cursor is not being reset because the job is timing out before we return state. (cc @taylordowns2000 )

When the queue clears tomorrow, @daissatou2 and I will try to run the job one form at a time to migrate historical submissions... and then reset the cursor to today's date to then run on an ongoing basis. And we will run the historical migration locally if taking too long on prod.

Thanks for the help!