singer-io / tap-kustomer

GNU Affero General Public License v3.0
2 stars 13 forks source link

Update sync.py to increase RESULT_RETURN_LIMIT #15

Closed dRuEFFECT closed 3 years ago

dRuEFFECT commented 3 years ago

As per comments in the code and first hand experience with eternal pagination as outlined in issue #12 , RESULT_RETURN_LIMIT needs to be increased for to account for hundreds of rows with the same updated_at timestamp. API docs say the max value for page size is 500, and testing done by @mascah proved that even setting a page size value of 1000 only returned 500 rows.

Description of change

(write a short description here or paste a link to JIRA)

QA steps

Risks

Rollback steps

mascah commented 3 years ago

I'm going to run this locally and try and do a complete extraction of all our data.

asaf-erlich commented 3 years ago

For both PR https://github.com/singer-io/tap-kustomer/pull/13 and https://github.com/singer-io/tap-kustomer/pull/15 we're happy to merge it, but looking at the long discussion on the referenced issue https://github.com/singer-io/tap-kustomer/issues/12 I'm not convinced this will solve the root cause. I will work on merging them and then try to provide guidance on what I believe will better address the root cause (based on the information discussed).

asaf-erlich commented 3 years ago

I've pushed a local branch to trigger circle to run on this one.

asaf-erlich commented 3 years ago

Looks like circle was broken due to some known python version and dependency compatibility issues, so I'm going to create a PR to fix circle.

asaf-erlich commented 3 years ago

https://github.com/singer-io/tap-kustomer/pull/18

asaf-erlich commented 3 years ago

Now that both circle fix PR and the other PR https://github.com/singer-io/tap-kustomer/pull/13 is merged I'm going to click the button to update this branch, then push the local branch again to trigger circle, once it's 💚 I will approve and merge this PR.

asaf-erlich commented 3 years ago

The build is green, merging it. I'll next comment on the github issue of what I really suspect is the root cause.