429 Too Many Requests #12

Open masardee opened 5 years ago

I used stitchdata.com to extract my Gitlab data and stored to other DB. There are > 100 projects defined in my gitlab.

But I always end up with "429 Too Many Requests" error when running the sync process.

This is my log :

2019-01-30 17:02:45,456Z    tap - INFO GET https://gitlab.com/api/v4/projects/XXXX/repository/commits?private_token=....................&page=27
2019-01-30 17:02:45,526Z    tap - INFO GET https://gitlab.com/api/v4/projects/XXXX/repository/commits?private_token=....................&page=28
2019-01-30 17:02:45,600Z    tap - INFO GET https://gitlab.com/api/v4/projects/XXXX/repository/commits?private_token=....................&page=29
2019-01-30 17:02:45,699Z    tap - INFO GET https://gitlab.com/api/v4/projects/XXXX/repository/commits?private_token=....................&page=30
2019-01-30 17:02:47,715Z    tap - CRITICAL Error making request to GitLab API: GET https://gitlab.com/api/v4/projects/XXXX/repository/commits?private_token=....................&page=30 [429 - b'<!DOCTYPE html>\n<html>\n<head>\n  <meta content="width=device-width, initial-scale=1, maximum-scale=1" name="viewport">\n  <title>429 Too Many Requests</title>

Any one can help?

Looks like you are being throttled by the GitLab.com API endpoint for making too many requests too quickly. I believe the throttle rate is calculated per minute, so if you're hammering "reload" during development you may be hitting that limit prematurely.

However, 30 pages of commits could be doing it on its own. If you can't get things done in one operation you have some options:

Get it done in batches. Modify your scraper to grab "at most 10 pages / X records" at a time, then merge the data downstream.
Throttle your client. Add a 429 error handler that sleeps for 60 seconds and retries. (I know, "adding magic sleeps" doesn't sound like the best answer...)
Don't use the API at all. In this case you're looking at commits, so you could make a local clone of the project and extract the data using a different method. You won't get API-throttled if you're querying a local filesystem, after all.

It sounds like you may be using a hosted service to do the API scraping, in which case you might not have direct access to the scraper's source code. In that case, you're going to have to engage somebody's support to find out what the best next steps are, since you either need to modify the behavior of the scraper (Stitchdata) or the behavior of the API it's connecting to (GitLab.com). Good luck.

PS - I'm just a user, I'm not especially knowledgeable about this particular codebase ... but a partially-informed answer to a question is better than no answers at all, right?

singer-io / tap-gitlab

429 Too Many Requests #12