Closed econti closed 2 years ago
Hi @econti, thanks for your contribution!
In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.
You did it @econti!
Thank you for signing the Singer Contribution License Agreement.
@dmosorast - Investigated a little further and figured out that the issue is caused by the same issue that was fixed for the Companies
resource in v0.5.1
. The Codat endpoint returns paginated results keyed under results
which is not checked properly for all paginated resources. This PR has the fix for all the relevant resources. Here are some logs that show the fix works for one example resource, Payments
:
Pre fix (0 records found for each company):
INFO Syncing payments for Company 1 of 37
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.496380090713501, "tags": {"endpoint": "payments", "http_status_code": 200, "status": "succeeded"}}
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"endpoint": "payments"}}
Post fix (records are now found):
INFO Syncing payments for Company 1 of 37
INFO Syncing page 1 for payments stream (company=<redacted>)
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.6119899749755859, "tags": {"endpoint": "payments", "http_status_code": 200, "status": "succeeded"}}
{"type": "RECORD", "stream": "payments", "record": {<redacted>}}
... (lots of records follow)
I went through and made sure the fix worked for all the streams that I modified as well.
Description of change
Currently any stream that is returned from a paginated endpoint is broken due to the wrong collection key being used. This fixes several resources by collecting results using the
results
key and using thePaginated
stream type.Manual QA steps
Risks
Rollback steps