search

turbot / steampipe-plugin-github

Use SQL to instantly query repositories, users, gists and more from GitHub. Open source CLI. No DB required.
https://hub.steampipe.io/plugins/turbot/github
Apache License 2.0
74 stars 30 forks source link

Struggling to use github_my_repository with a lot of repos (50k+) #461

Open chris-garrett opened 2 weeks ago

chris-garrett commented 2 weeks ago

Describe the bug Running select * from github.github_my_repository causes timeouts and rate limits: 

SQL Error [HV000]: ERROR: rpc error: code = Unknown desc = github: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to

Steampipe version (steampipe -v) Steampipe v1.0.0

Plugin version (steampipe plugin list)

+-----------------------------------------------+---------+-------------+
| Installed                                     | Version | Connections |
+-----------------------------------------------+---------+-------------+
| hub.steampipe.io/plugins/turbot/github@latest | 1.0.0   | github      |
| hub.steampipe.io/plugins/turbot/jira@latest   | 1.0.0   | jira        |
+-----------------------------------------------+---------+-------------+

To reproduce Run the command above.

Expected behavior Results are returned from query.

Additional context I did a bit of poking around and I think that the plugin is pulling a large number of forks that I have nothing to do with. I'm attached to the EpicGames repo and it looks like it's trying to pull 50k+ forks from there?

I'm also surprised that I'm hitting rate limits at all. There is a well known 5k request per hour limit I would have expected this plugin to use that limit as a sane default.