search

phillipj / gitlab-search

Command line tool to search for contents in GitLab repositories
MIT License
296 stars 35 forks source link

GitLab Rate Limit #30

Open EvilJordan opened 2 years ago

EvilJordan commented 2 years ago

The GitLab.com docs (https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits) provide a 10 request per minute rate limit for "Advanced, project, and group search API (for a given IP address)"

So, user-beware. If you have lots of repos/groups/projects whatever they're called, you're gonna have a bad time.

martinbutt commented 2 years ago

Not a direct solution, but when connecting over HTTPS, you can limit the number of threads using the --concurrency flag. See https://github.com/phillipj/gitlab-search/pull/21#issuecomment-812909745 - you might still hit the rate limit, but this solved the issue for me.

EvilJordan commented 2 years ago

This is a different issue, different rate limit, but I appreciate the guidance!

phillipj commented 2 years ago

Buhu, that's too bad as that pretty low. Thanks for sharing @EvilJordan 👍🏻

graham73may commented 2 years ago

Came here with the same issue. Does this basically render the tool unusable? 😢

EvilJordan commented 2 years ago

It wasn't sufficient for me, so I built this not-perfect-but-got-it-done instead. I'm reconstructing this from my notes, which appear to be incomplete, so apologies if it only gets you 90% of the way there.

I manually visit my endpoint URL, substituting PAGE_NUMBER with 1 through N, until the return is empty. This could be automated, but I was in a hurry and didn't want to fuss: 

https://gitlab.com/api/v4/groups/{GROUP_NUMBER}/projects?include_subgroups=true&per_page=300&page={PAGE_NUMBER}

Next, I combine all results of the above queries into a, comma separated, (repos.json) file, with everything separated by a comma.

Then, run the below bash command to clone everything (relies on jq being available and if you have projects named the same thing, this loop will not be thrilled):

for repo in $(jq ".[].ssh_url_to_repo" repos.json | tr -d '"'); do git clone $repo; done;
ldbglobe commented 2 years ago

Due to my lack of knowledge of ReScript I wrote an really simple alternative based on a NodeJS Gitlab REST API package. The main objective was to achieve my search across a bit more than 100. So my work is quite straightforward.

https://github.com/ldbglobe/gitlab-finder

scotteuser commented 2 years ago

Thanks @ldbglobe - that seems to work great, much appreciated!

phillipj commented 2 years ago

Very cool indeed @ldbglobe 👏

On Mon, Feb 21, 2022 at 9:54 AM Scott Euser @.***> wrote:

Thanks @ldbglobe https://github.com/ldbglobe - that seems to work great, much appreciated!

— Reply to this email directly, view it on GitHub https://github.com/phillipj/gitlab-search/issues/30#issuecomment-1046617098, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJMWE6N3EP3L2O2ULFUMYDU4H4T3ANCNFSM5KBLYILQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>