singer-io / tap-pipedrive

A Singer.io tap for extracting data from the Pipedrive API

GNU Affero General Public License v3.0

13 stars 34 forks source link

Retry null requests for DealsFlow stream #131

Closed leslievandemark closed 1 year ago

leslievandemark commented 1 year ago

Description of change

Adds retries for dealflows requests that return a status of 200 but a null body.
Adds the 'Accept-Encoding': 'application/json' header to all requests, which is advised in pipedrives docs: https://pipedrive.readme.io/docs/core-api-concepts-requests

Manual QA steps

applied the change as a pr-alpha to client in TDL-22119 and resolved the error they were receiving.

Risks

Adding a new header could possibly change things for other streams, but the tap expects data to be returned as json, so it would likely be for the better.

Rollback steps

revert this branch

GeorgeTouros commented 1 year ago

This change seems to have broken our data pipeline using Stitch. Since March 12, we have been experiencing infinite retries on the object, returning NULL values and status 200, causing the pipeline to continue running in perpetuity. See the latency logs attached.

The max number of retries seems to also not work here, because every few times we get a response with 1 row, thus re-initiating the cycle all along.