search

singer-io / tap-quickbase

Singer.io tap for Quick Base API
GNU Affero General Public License v3.0
7 stars 3 forks source link

Add error handling and backoff when making requests to Quick Base #20

Open wkirby007 opened 6 years ago

wkirby007 commented 6 years ago

Current Summary: We don't have any error handling around making requests to Quick Base which exposes us to blips in the SSL handshake which are apparently a known issue.

Adding backoff to the request code should improve the situation.

Hi.

We've just started using this integration into Quick Base from Stitchdata. Within in QB, we have multiple applications, but we only want the data from one of them. We setup a user token with access to the singular app and have used that app reference in the integration.

I can see from the logs that the tap receives all the apps, but only structures the streams for those that relate to the app we are interested in. However, we get error reports as it shows the below in the logs, but it actually appears to relate to the other apps in the QB realm which the user token will not have access too?

HTTPSConnectionPool(host='pulse.quickbase.com', port=443): Max retries exceeded with url: /db/bmpvry5t7?options=num-100&query=%7B2.AF.1532359647000%7D&act=API_DoQuery&includeRids=1&clist=23.2.24.6.16.48.55.31.25.54.12.38.32.18.3.21.14.1.26.44.42.46.43.41.45.36.13.33.47.49.5.4.20&slist=2&fmt=structured&realmhost=&usertoken=....................................... (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:645)'),))
emlama commented 6 years ago

Hi Kirby,

We do have a known issue where our API will sometimes have an SSLHandshake error and you'll typically have to wait a few seconds and then retry. Is there any backoff for the API calls here?

Best,

wkirby007 commented 6 years ago

Thanks for the reply.

We are currently backing off to PostgreSQL. The experience we have with the above error however coincides with not all our tables being created and populated? We see a single table of data only in this instance but the stream would suggest, as does the QV application itself, that more tables should be processed.

Sent from my iPhone

On 8 Nov 2018, at 03:30, Matthew Saforrian notifications@github.com wrote:

Hi Kirby,

We do have a known issue where our API will sometimes have an SSLHandshake error and you'll typically have to wait a few seconds and then retry. Is there any backoff for the API calls here?

Best,

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

emlama commented 6 years ago

I also encountered this issue using the Stitch platform. I did observe that the Quick Base connector seems to be having an issue with how it's iterating through all of the records. I'm seeing it as trying to load 544,500 records but only have 4,448 records in that table. I suspect that it's not incrementing the counter for how many records to retrieve.

I wonder if this SSLhandshake is being caused by too many fetches.

Anyhow, where can I see the retry logic for requests being made into Quick Base? That would be a good place to start.

wkirby007 commented 6 years ago

Thanks again for the response. I've downloaded the logs and attached if that is what you are referring to? Please let me know if there is something else I can capture to assist.

Thanks

Will

On 8 Nov 2018, at 15:45, Matthew Saforrian notifications@github.com wrote:

I also encountered this issue using the Stitch platform. I did observe that the Quick Base connector seems to be having an issue with how it's iterating through all of the records. I'm seeing it as trying to load 544,500 records but only have 4,448 records in that table. I suspect that it's not incrementing the counter for how many records to retrieve.

I wonder if this SSLhandshake is being caused by too many fetches.

Anyhow, where can I see the retry logic for requests being made into Quick Base? That would be a good place to start.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/singer-io/tap-quickbase/issues/20#issuecomment-437043006, or mute the thread https://github.com/notifications/unsubscribe-auth/AMQS_XYRj-nKVBROSjekLwvqHKGfD7Caks5utFG0gaJpZM4YQwVK.

timvisher commented 6 years ago

@msaforrian At the moment it doesn't look like we're doing any error handling or backoff when we make requests to Quick Base. Adding those should be fairly easy.

We've had good success using the backoff module for this purpose. Here's a decently complex example of doing so.

Would either of you want to take a stab at utilizing it to make this error handling better?

WaverlyK commented 1 year ago

Wondering if there is any update on this?