Closed dRuEFFECT closed 3 years ago
I'm going to run this locally and try and do a complete extraction of all our data.
For both PR https://github.com/singer-io/tap-kustomer/pull/13 and https://github.com/singer-io/tap-kustomer/pull/15 we're happy to merge it, but looking at the long discussion on the referenced issue https://github.com/singer-io/tap-kustomer/issues/12 I'm not convinced this will solve the root cause. I will work on merging them and then try to provide guidance on what I believe will better address the root cause (based on the information discussed).
I've pushed a local branch to trigger circle to run on this one.
Looks like circle was broken due to some known python version and dependency compatibility issues, so I'm going to create a PR to fix circle.
Now that both circle fix PR and the other PR https://github.com/singer-io/tap-kustomer/pull/13 is merged I'm going to click the button to update this branch, then push the local branch again to trigger circle, once it's 💚 I will approve and merge this PR.
The build is green, merging it. I'll next comment on the github issue of what I really suspect is the root cause.
As per comments in the code and first hand experience with eternal pagination as outlined in issue #12 , RESULT_RETURN_LIMIT needs to be increased for to account for hundreds of rows with the same updated_at timestamp. API docs say the max value for page size is 500, and testing done by @mascah proved that even setting a page size value of 1000 only returned 500 rows.
Description of change
(write a short description here or paste a link to JIRA)
QA steps
Risks
Rollback steps