singer-io / tap-codat

A Singer.io tap to extract data from the Codat API
GNU Affero General Public License v3.0
2 stars 7 forks source link

Fix broken paginated streams #18

Closed econti closed 2 years ago

econti commented 2 years ago

Description of change

Currently any stream that is returned from a paginated endpoint is broken due to the wrong collection key being used. This fixes several resources by collecting results using the results key and using the Paginated stream type.

Manual QA steps

Risks

Rollback steps

singer-bot commented 2 years ago

Hi @econti, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

singer-bot commented 2 years ago

You did it @econti!

Thank you for signing the Singer Contribution License Agreement.

econti commented 2 years ago

@dmosorast - Investigated a little further and figured out that the issue is caused by the same issue that was fixed for the Companies resource in v0.5.1. The Codat endpoint returns paginated results keyed under results which is not checked properly for all paginated resources. This PR has the fix for all the relevant resources. Here are some logs that show the fix works for one example resource, Payments:

Pre fix (0 records found for each company):

INFO Syncing payments for Company 1 of 37
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.496380090713501, "tags": {"endpoint": "payments", "http_status_code": 200, "status": "succeeded"}}
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"endpoint": "payments"}}

Post fix (records are now found):

INFO Syncing payments for Company 1 of 37
INFO Syncing page 1 for payments stream (company=<redacted>)
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.6119899749755859, "tags": {"endpoint": "payments", "http_status_code": 200, "status": "succeeded"}}
{"type": "RECORD", "stream": "payments", "record": {<redacted>}}
... (lots of records follow)

I went through and made sure the fix worked for all the streams that I modified as well.