search

singer-io / tap-asana

Singer.io tap for extracting Asana data
GNU Affero General Public License v3.0
7 stars 20 forks source link

TDL-15274: Fix pagination for Stories and Tasks #40

Closed hpatel41 closed 2 years ago

hpatel41 commented 3 years ago

Description of change

TDL-15274: Fix pagination for Stories and Tasks

Manual QA steps

Risks

Rollback steps

hpatel41 commented 2 years ago

Project is also used for section stream, please add this same logic for section as well: https://github.com/singer-io/tap-asana/blob/master/tap_asana/streams/sections.py

Added caching of Project IDs for the section stream.

hpatel41 commented 2 years ago

Can this issue be reproduced via tap-tester test?

@kspeer825 We cannot reproduce this issue in the tap-tester, as in the first loop we get the projects on which we need to fetch the required data, based on the project ids.

To fetch the data for the next page we require an offset token which is valid for 15-20 minutes. Hence, the inner loops should have enough data to keep iterating over the project ids before fetching the next page for projects.

Reference: https://developers.asana.com/docs/pagination

hpatel41 commented 2 years ago

Please merge the regression suite PR #39 to master which is approved. And then merge master back into this PR. I would like to understand why we do not see the standard pagination test fail for this issue. We should be able to write a failing tap-tester test for this case and then see it pass with the code fix here. I do not think this should be merged until that is done.

@kspeer825 We require more data for failing the tests, but for 3000 records, the tap takes 1.5 minutes. Hence, to make the loop iterate for around 15 minutes, we will need to generate a huge amount of data. So, we are working to generate more data to fail the test cases.