search

singer-io / tap-freshdesk

GNU Affero General Public License v3.0
15 stars 30 forks source link

Extremely slow data load #47

Open thomas-spotnana opened 2 years ago

thomas-spotnana commented 2 years ago

I am using tap-freshdesk with Stitch to sync data from Freshdesk to Snowflake. The integration tool seems to load only around 250 rows per 10 minutes, which is extremely slow. This makes the tool non-usable. I suspect this is due to the way data is batched, but was hoping you could elaborate/fix this so that loads from Freshdesk can be as fast as other tools.

shubham-124 commented 2 years ago

can anyone help or update here! I am also facing similar issue.

jaketho commented 1 year ago

The issue with rate limiting is here https://github.com/singer-io/tap-freshdesk/blob/dc64f26689fca24d9307fd2b075f523a892f1d73/tap_freshdesk/__init__.py#L43

According to the documentation, rate limiting is "x requests per y seconds". I think the default of (1, 2) assumes the growth plan with a limit of 50 requests per minute and allows some headroom for other API calls. Depending on your plan and plan age, you can determine your actual rate limit and adjust accordingly. You can find Freshdesk API limits for current plans here > https://support.freshdesk.com/en/support/solutions/articles/225439-what-are-the-rate-limits-for-the-api-calls-to-freshdesk- For older plans, reference the image at the bottom of this page > https://support.freshservice.com/en/support/solutions/articles/50000000293-what-is-the-rate-limit-for-apis-across-all-plans-

To avoid problems raised in other issues, instead of using the full amount of requests per time, like (50, 60), I use (1, 1.2).

A possible permanent solution for this issue would be to look at the x-requests-total and assume the plan and rate limit or add as a configuration parameter.