search

OpenFn / Legacy-OpenFn-iPaaS-v0

OpenFn Integration Platform
https://www.openfn.org
Other
3 stars 4 forks source link

ODK Submissions not pulling through #400

Closed stuartc closed 8 years ago

stuartc commented 8 years ago

When there is a substantial amount of submissions on a given ODK Aggregate instance, the periodic importer does not find any new submissions.

The issue is believed to be caused by the ODK cursor's time stamp being reset in: https://github.com/OpenFn/OpenFn-Site/blob/master/app/workers/odk_sf_legacy/odk_to_salesforce/odk.rb#L53

ODK appears to walk the submissions list from a given point (the cursor), and stops after numEntries has been hit. Since all queries are expected to honour that all submissions are filed in order; the current implementation has a skewed offset into the dataset.

stuartc commented 8 years ago

@taylordowns2000, I've push the changes to develop, master and it's on production.

What I've done is let the periodic import run (which updated the cursor to the last one it got). I think ran the importer by hand for the mappings you reported had an issue.

I can verify that 2/3 mappings received new submissions, so feeling confident this is gonna make things a lot better.

Feel free to close if you can validate. :)

taylordowns2000 commented 8 years ago

Thank you Stu, this is fantastic.

Which mapping did not get new submissions?

Taylor

On Mon, 19 Oct 2015 at 12:03 Stuart Corbishley notifications@github.com wrote:

@taylordowns2000 https://github.com/taylordowns2000, I've push the changes to develop, master and it's on production.

What I've done is let the periodic import run (which updated the cursor to the last one it got). I think ran the importer by hand for the mappings you reported had an issue.

I can verify that 2/3 mappings received new submissions, so feeling confident this is gonna make things a lot better.

Feel free to close if you can validate. :)

— Reply to this email directly or view it on GitHub https://github.com/OpenFn/OpenFn-Site/issues/400#issuecomment-149259552.

stuartc commented 8 years ago

I stand to be corrected, by I believe it was 487.

taylordowns2000 commented 8 years ago

Yeah, that looks right. I got a hold of their aggregate data and that form hasn't been used in a while. Also it seems as though anew submission was pulled via auto-fetch for the other two forms since then, which is good news. Thanks for your quick work on this.

On Tue, 20 Oct 2015 at 04:18 Stuart Corbishley notifications@github.com wrote:

I stand to be corrected, by I believe it was 487.

— Reply to this email directly or view it on GitHub https://github.com/OpenFn/OpenFn-Site/issues/400#issuecomment-149474037.