Open electriquo opened 7 months ago
@cbruno10 maybe you could add more information
Hello, @electriquo. Sorry for the delayed response.
Before we proceed with the fixes, I have a couple of questions, could you please take a look?
I attempted to replicate the error by querying the github_issue
and github_pull_request
tables, but I was unable to reproduce it. I tried the query select count(*) from github_workflow where repository_full_name in (select name_with_owner from github_my_repository)
multiple times, this query returns the result successfully all the time.
+-------+
| count |
+-------+
| 893 |
+-------+
Thank you!
@ParthaI Install the github modules (steampipe-mod-github-compliance, steampipe-mod-github-insights, steampipe-mod-github-sherlock) and navigate to the GitHub Default Branch Protection Report.
I have more than 2000 active (non-archived) repositories.
Thanks, @electriquo, for the detailed information, I will give it a try to reproduce the issues, and make the necessary changes.
Hello, @electriquo, following our previous discussion, I set up my local environment to try and replicate the error.
Despite my attempts, I couldn't reproduce the rate limit error. The compliance/insight mods are functioning correctly across approximately 370
repositories in our organization.
For your information, the plugin initially used the REST API to generate results, which was prone to errors, particularly those related to rate limits. As a remedy, we've implemented GraphQL query support.
Additionally, I experimented with the plugin code in my local environment, making around 2300
API calls, and it worked without issues, completing in 1,671.1
seconds.
Query Result:
> select count(*) from github_my_repository
+-------+
| count |
+-------+
| 11871 |
+-------+
Time: 1,671.1s. Rows fetched: 11,871. Hydrate calls: 0.
Thanks!
@ParthaI You were experimenting, which is a reproduction best effort. Yet, it does not say that there is no issue :) Which information/logs do you need to get to the bottom of it?
As you can see in the description, the rate limit error is clear.
Hello @electriquo,
Thank you for your patience and for highlighting the issue once more. Indeed, the rate-limit error you mentioned is a significant concern, when navigating to the GitHub Default Branch Protection Report, given the extensive number of active (non-archived) repositories you manage, exceeding 2000.
In an effort to meticulously replicate this scenario, I conducted tests involving over 2000 API calls within a single query. Regrettably, these tests did not trigger the same rate-limit error, which suggests the situation might be influenced by specific conditions or a huge number of repositories in the organization.
Understanding the importance of accurately diagnosing and resolving this issue, I'd like to delve deeper. The dashboard you referred to leverages the github_my_repository
table to fetch and display data. To assist us further in isolating the cause, could you kindly confirm if executing the query select * from github_my_repository
directly in your Steampipe environment results in the same rate-limit error? This information will be invaluable in our investigation.
Your cooperation and insights are greatly appreciated!
@ParthaI
To assist us further in isolating the cause, could you kindly confirm if executing the query
select * from github_my_repository
directly in your Steampipe environment results in the same rate-limit error?
No issue here but when I open my-steampipe-host.com/github_insights.dashboard.default_branch_protection_report
I get the limit-rate error.
Thank you, @electriquo, for conducting those tests and sharing your findings. Replicating the rate-limit error appears to be challenging with less set of data set in our environment.
According to the documentation, using a Personal Access Token (PAT) allows for 5000
API calls within a certain timeframe. However, running the GitHub Insight Mods
seems to exceed this limit in your setup, likely due to the large number of repositories (around 2000) in your environment.
In comparison, our environment hosts approximately 330
repositories. Before executing the GraphQL query, our initial rate limit was 5000
. After executing select count(*) from github_my_repository
, this limit dropped to 4923
, indicating that approximately 77
API calls were made (per page result limit value is set to 5
) when querying the github_my_repository
table.
Here's the GraphQL query to check the rate limit:
query {
viewer {
login
}
rateLimit {
limit
remaining
used
resetAt
}
}
You can find more details on how rate limits are calculated for GraphQL queries in the GitHub documentation.
Upon reviewing the GitHub insights mod, specifically the github_insights.dashboard.default_branch_protection_report
, it seems the github_my_repository
table is queried multiple times, which could contribute to reaching the rate limit.
I'm further investigating the plugin and the insight mod to understand this behavior better and will update you with any progress.
Thank you once more for your assistance!
using a Personal Access Token (PAT) allows for 5000 API calls within a certain timeframe.
To clear any doubts, I am using a dedicate PAT; meaning that only Steampipe is using this PAT. Thus, the rate-limit must come when using Steampipe only.
Could it be that not only the amount of repositories are the factor but the amount of metadata (such as a big Git history/commits)?
Could it be that not only the amount of repositories are the factor but the amount of metadata (such as a big Git history/commits)?
Absolutely, @electriquo, you're correct. However, it's worth noting that within the Steampipe Dashboard, API calls are not initiated until you actively click on any of the hyperlinks.
My repositories are reach, they are big with many metadata. How do you advise to address it or is this plugin is only for a small scale and thin repositories?
Hello @electriquo,
I hope this message finds you well. I am currently working on this issue aimed at reducing the number of API calls while quering the table github_my_repository
I have made a few changes in the GitHub Plugin
and the GitHub Insight Mod
. Unfortunately, I have yet to finalize an effective solution and find myself in need of a bit more information. When you have a moment, could you kindly assist me by checking on a few details?
Could you please share how long the query select * from github_my_repository
takes to execute in your current setup? If you could follow these steps, it would be immensely helpful:
steampipe query
..timing on
.select * from github_my_repository
.I have recently committed some updates to the GitHub plugin repository on the branch update-limit-for-github-my-repo-table
and to the GitHub Insight mod on the branch updated-query-to-hit-cache
. Could I trouble you to fetch these updates to your local setup and build the plugin from there?
Please note, these modifications are specifically for the GitHub Default Branch Protection Report
dashboard. My goal is to ensure these changes perform effectively, especially with larger datasets.
If you're willing to test the scenario, here are the steps:
git clone https://github.com/turbot/steampipe-plugin-github.git
.cd steampipe-plugin-github
.git checkout update-limit-for-github-my-repo-table
.make
.steampipe service start
.git clone https://github.com/turbot/steampipe-mod-github-insights.git
.cd steampipe-mod-github-insights
.git checkout updated-query-to-hit-cache
.powerpipe server
.Your feedback on these changes would be invaluable to me.
Thank you very much for your cooperation and assistance in this matter.
Hi @electriquo, have you had the chance to try it out yet?
@ParthaI Sorry, will try it soon.
Could you please share how long the query select * from github_my_repository takes to execute in your current setup?
$ steampipe query
Welcome to Steampipe v0.22.1
For more information, type .help
> .timing on
> select * from github_my_repository
Error: github: non-200 OK status code: 403 Forbidden body: "{\n \"documentation_url\": \"https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 279A:5B4F:18264BC:2CAC1C2:65F97C91.\"\n}" (SQLSTATE HV000)
Time: 47.1s. Rows fetched: 250. Hydrate calls: 17,250.
Could I trouble you to fetch these updates to your local setup and build the plugin from there? ... here are the steps:
Where is the install command for update-limit-for-github-my-repo-table
plugin and updated-query-to-hit-cache
mod?
What am I missing?
$ steampipe query Welcome to Steampipe v0.22.1 For more information, type .help >.timing on >select * from github_my_repository
Error: github: non-200 OK status code: 403 Forbidden body: "{\n \"documentation_url\": \"https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 279A:5B4F:18264BC:2CAC1C2:65F97C91.\"\n}" (SQLSTATE HV000)
Time: 47.1s. Rows fetched: 250. Hydrate calls: 17,250.
I believe as per our previous conversation it was working fine.
Where is the install command for
update-limit-for-github-my-repo-table
plugin andupdated-query-to-hit-cache mod
? What am I missing?
Please follow the following steps:
Steampie Plugin changes:
git clone https://github.com/turbot/steampipe-plugin-github.git
.cd steampipe-plugin-github
.git checkout update-limit-for-github-my-repo-table
.make
.steampipe service start
.Steampipe Github Insights Mod:
git clone https://github.com/turbot/steampipe-mod-github-insights.git
.cd steampipe-mod-github-insights
.git checkout updated-query-to-hit-cache
.powerpipe server
.@ParthaI Haven't forgot, didn't find time this :(
Hey @electriquo , I was able to reproduce the behaviours you were seeing (albeit with a much lower repository count) with select * from github_my_repository
.
I looked into the plugin code, and found our error handling code does retry secondary rate limit errors (https://github.com/turbot/steampipe-plugin-github/blob/ec932825c781a66c325fdbc5560f96cac272e64f/github/errors.go#L39-L43), with the plugin's retry config set to use an exponential backoff algorithm with max 10 retries (https://github.com/turbot/steampipe-plugin-github/blob/ec932825c781a66c325fdbc5560f96cac272e64f/github/errors.go#L54-L62); this is probably still too aggressive based on how many API calls we'd be making for that many repos.
If you're directly running a query, one suggestion is to only select the columns you want to retrieve.
If you're running a mod dashboard/benchmark/control, you could also try using rate limiters to slow down Steampipe.
For instance, you can try adding this into your github.spc
:
# stay well under the 100 hydrate/list/get functions concurrently based on limits in https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#secondary-rate-limits
plugin "github" {
limiter "github_global_concurrency" {
max_concurrency = 30
}
}
And then see if that helps with running the dashboards or benchmarks you were trying before.
I think select * from my_repository
will still be tough for the plugin to complete even with rate limiters though based on how GitHub checks for secondary rate limits, as they have some internal timeouts and we request a lot of data as part of the query when selecting all columns.
For instance, with the limiter settings above, I avoided getting the secondary abuse rate limits, but eventually got this unhelpful 502 Bad Gateway error from them with no additional information:
Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug.
I don't believe a lot of our dashboards run this type of query though.
@cbruno10
found our error handling code does retry secondary rate limit errors
Does it mean that you are working on a fix and I should wait for a new release?
For instance, you can try adding this into your
github.spc
:# stay well under the 100 hydrate/list/get functions concurrently based on limits in https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#secondary-rate-limits plugin "github" { limiter "github_global_concurrency" { max_concurrency = 30 } }
Where in the docs did you find about github_global_concurrency
and max_concurrency?
@electriquo
We are looking into if we can improve the error handling for rate limit errors, but we're not actively working on an identified fix and don't have a schedule on when it will be released.
I'd suggest still trying to use a limiter
and seeing if that helps your executions.
In the example I sent, the github_global_concurrency
is the name I gave the limiter resource, but general documentation can be found at https://steampipe.io/docs/guides/limiter.
@cbruno10
We are looking into if we can improve the error handling for rate limit errors, but we're not actively working on an identified fix and don't have a schedule on when it will be released.
Thanks
In the example I sent, the
github_global_concurrency
is the name I gave the limiter resource, but general documentation can be found at steampipe.io/docs/guides/limiter.
In the future, how one may know the variables that can be set in a plugin to handle concurrency and rate limiting? If you would not have mentioned the structure and the variable, I guess that most won't know about it :| That's why I asked
Where in the docs did you find about
github_global_concurrency
and max_concurrency?
Maybe it should be documented in steampipe.io/docs/guides/limiter.
I was able to reproduce the behaviours
you can try adding this into your
github.spc
:# stay well under the 100 hydrate/list/get functions concurrently based on limits in https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#secondary-rate-limits plugin "github" { limiter "github_global_concurrency" { max_concurrency = 30 } }
If you apply the block in github.spc
, did it mitigate the rate-limiting which you were able to reproduce?
@electriquo With that block in my github.spc
, yes I did see the rate-limiting was resolved. My query execution time increased, but I was able to get all of the data for my repos (though this number is much lower than the amount of repos you have).
Were you able to give the limiter a try, either with max concurrency of 30 (or a different number)? If so, did you see any more consistency in getting results back?
Also, we should probably mention rate limiters and at least link to the doc on steampipe.io, which has some examples, so we'll add that to our backlog to see where that section belongs.
Were you able to give the limiter a try, either with max concurrency of 30 (or a different number)?
It is on my list for tomorrow :)
If so, did you see any more consistency in getting results back?
Promise to keep you posted
Also, we should probably mention rate limiters and at least link to the doc on steampipe.io, which has some examples, so we'll add that to our backlog to see where that section belongs.
Awesome, thanks you :)
@cbruno10
$ steampipe --version
Steampipe v0.22.2
$ steampipe plugin list --output json | jq -r '.installed[].name'
hub.steampipe.io/plugins/turbot/github@0.39.1
$ powerpipe mod list --output json | jq -r '.[].dependency_path'
github.com/turbot/steampipe-mod-github-insights@v0.4.0
github.com/turbot/steampipe-mod-github-compliance@v0.7.0
github.com/turbot/steampipe-mod-github-sherlock@v0.13.0
$ cat ~/.steampipe/config/github.spc
connection "github" {
plugin = "hub.steampipe.io/plugins/turbot/github@v0.39.1"
}
plugin "github" {
limiter "github_global_concurrency" {
max_concurrency = 30
}
}
When I navigate to the GitHub Default Branch Protection Report dashboard, the rate-limit error still pops.
But if I repeat https://github.com/turbot/steampipe-plugin-github/issues/398#issuecomment-2006990182 does not return a rate-limit error message
$ steampipe query
Welcome to Steampipe v0.22.2
For more information, type .help
> .timing on
> select count(*) from github_my_repository
...
Time: 7.6s. Rows fetched: 1,088. Hydrate calls: 0.
And then navigating to the GitHub Default Branch Protection Report dashboard, the rate-limit error does not pops but besides the repository list, all other columns are empty :(
I look at look at the plugin log, and I found that it seems not to use (but do honor) the max concurrency connections. Here are some log lines
2024-04-10 07:16:43.590 UTC [INFO] waitForPluginLoad: hub.steampipe.io/plugins/turbot/github@0.39.1 is now loaded and ready (0x40007cd880)
2024-04-10 07:16:43.593 UTC [INFO] steampipe-plugin-github.plugin: [INFO] 1712733403165: Plugin execute table name: github_my_repository quals: NONE (1712733403165)
2024-04-10 07:16:43.593 UTC [INFO] steampipe-plugin-github.plugin: [INFO] 1712733403165: Executing for 1 connection: github
2024-04-10 07:16:43.593 UTC [INFO] steampipe-plugin-github.plugin: [INFO] 1712733403165: Setting max concurrent connections to 25
2024-04-10 07:16:43.593 UTC [INFO] steampipe-plugin-github.plugin: [INFO] 1712733403165: resolveRateLimiterScopeValues (github-1712733403165)
2024-04-10 07:16:43.593 UTC [INFO] steampipe-plugin-github.plugin: [INFO] 1712733403165: resolvedRateLimiterConfig: no rate limiters (github-1712733403165)
2024-04-10 07:16:51.016 UTC [INFO] steampipe-plugin-github.plugin: [INFO] 1712733403520: rowData chan returned nil - wait for rows to complete (github-1712733403165)
2024-04-10 07:16:51.017 UTC [INFO] steampipe-plugin-github.plugin: [INFO] 1712733403520: streamRows - nil row, stop streaming (github-1712733403165)
2024-04-10 07:16:51.023 UTC [INFO] steampipe-plugin-github.plugin: [INFO] 1712733403520: Plugin execute table: github_my_repository closing error chan and output chan (1712733403165)
Also tried with max_concurrency = 10
, to check whether it will still be honored and/or resolve the issue by examining the log file and the dashboard, but everything was the same :(
@cbruno10 Hi, any insights?
@cbruno10 Hello, do you have any new insights?
@cbruno10 @ParthaI Hello, did we lost all momentum?
Apologies for the radio silence on this issue @electriquo !!
I think the reason why you didn't see the limiter tags getting honored could be indicative of a caching issue.
Could you please retry the queries by launching a fresh instance of Steampipe and killing any old instances?
Command that you could try before running steampipe query
:
steampipe service stop --force
pkill -f steampipe
Could you please retry the queries by launching a fresh instance of Steampipe and killing any old instances?
@misraved That's what I always did :)
I think the reason why you didn't see the limiter tags getting honored could be indicative of a caching issue.
Seems like firing in all directions rather than base things on data. Can we get to the bottom of it or should I just drop the GitHub plugin as a whole?
Thanks for the clarification @electriquo !!
We are actively exploring options to enhance the rate-limiting capabilities within our plugin. Currently, there is a limited selection of solutions that offer a straightforward method for managing errors originating from the API.
@misraved Could you please follow https://github.com/turbot/steampipe-plugin-github/issues/398#issuecomment-2046680830?
Hi @electriquo , sorry for losing track of this thread.
select * from steampipe_plugin_limiter
(taken from Viewing and Overriding Limiters). If you run that query, are you able to see your limiter with the correct settings?@cbruno10
The rate-limit error should be handled generally regardless the component that causes it, especially when there is a clear protocol for rate-limit. Specifically, GitHub has Rate limits for the REST API.
are you able to see your limiter with the correct settings?
No.
I followed your guidance to set github_global_concurrency
and executed the query. As you can see below, GitHub is not in the list.
$ steampipe query "select distinct name from steampipe_plugin_limiter"
+---------------------------------------------------+
| name |
+---------------------------------------------------+
| aws_servicequotas_list_service_quotas |
| aws_servicequotas_list_aws_default_service_quotas |
| aws_servicequotas_list_tags_for_resource |
+---------------------------------------------------+
Why would you expect to see the limiter appear here when the plugin log clearly states so?
Dashboards in general run a lot of queries simultaneously, so it'll be more common to hit rate limiters when running dashboards
From your words, I understand that dashboards are useless :(
If running dashboards, which ones have you tried beside GitHub Default Branch Protection Report? And have they all given you errors?
Received the same error across few GitHub's dashboards.
@electriquo
Looking at your limiter configurations, can you please update it to:
plugin "hub.steampipe.io/plugins/turbot/github@v0.39.1" {
limiter "github_global_concurrency" {
max_concurrency = 30
}
}
Since you're using a specific plugin version, the plugin
reference needs to reference that version.
Afterward, can you please restart Steampipe, steampipe service restart
and then run select * from steampipe_plugin_limiter
and see if the limiter then shows up?
If it does, can you please try running the single query you shared above first and see if that still executes correctly? If so, can you please then run the dashboard again? You may still get throttling errors, and if so, can you please try lowering max_concurrency
to 20 (and then 10)?
Thanks!
@cbruno10 Although I am cooperating, I am not the Steampipe QA team :) I've done whatever I was asked until now, but feel that there is firing to all directions in hope to hit something, and Steampipe do not need me for it - Steampipe team can and should reproduce everything on their own to save this back and forth writing.
Since you're using a specific plugin version, the plugin reference needs to reference that version.
This doesn't sound correct for me, just another shot in the dark.
After you took the toll to create and environment to:
I'd be happy to continue and assist.
@electriquo Thanks for all of your testing efforts and sharing so far, we appreciate all of the info you've shared! We try to reproduce users' environments as much as we can, but sometimes that's difficult as we don't have direct access to see how many repositories, branches, users, branch protection settings, etc., that are in those environments, so we're not able to reproduce them exactly. We do have a GitHub organization with a fairly large number of repositories, but definitely not as large as the one you're querying, so any information you're able to share is helpful for us.
In regards to the rate limiter, I was able to reproduce the rate limiter not loading when a specific plugin version was installed, but once I installed GitHub plugin v0.39.1 and created a limiter I shared above:
plugin "hub.steampipe.io/plugins/turbot/github@v0.39.1" {
limiter "github_global_concurrency" {
max_concurrency = 30
}
}
Then I could see my limiter being loaded and used by the plugin.
I was able to reproduce the rate limiter not loading when a specific plugin version was installed
This holds. Had to set the max_concurrency
to 1.
I only wonder why does the version must be specified when the same file already declares which plugin (and version) to use.
I ran into this issue today. I have access to 3800 repos and 10 orgs but I’m only interested in a handful of those orgs for querying.
I haven't tried the limiter yet but my first inclination was to see if there was any way to scope down the repos/orgs that get queried in the config file, similar to how the AWS plugin handles it.
I think that coupled with the limiter would cover my use case.
@jmreicha No matter what I did, I couldn't make this work as I was always facing the rate limit.
Hi @jmreicha , are you running standalone queries or benchmarks/dashboards from mods? If running queries, there are ways to structure the queries sometimes to reduce the number of requests and work better within the rate limit. For the mods, I think we can add better filtering capabilities to optimize the queries, so if there are specific ones you're interested in (along with any you're interested in @electriquo ), please let us know and we can have a look at those first.
Hi @cbruno10, Thank you.
I cannot tell what would I like to see since I cannot explore the plugin/mods due to the rate limit. Thus, I cannot share much about the usage
if there are specific ones you're interested in, please let us know and we can have a look at those first.
Hey @cbruno10
are you running standalone queries or benchmarks/dashboards from mods?
I noticed the issue when I was playing around with dashboards, specifically the organization best practices. That would be my main use case probably. I was also triggering the rate limit pretty easily doing queries but I will play around with that more and see if I can make better queries.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
Is it really stale or there is no fix?
Describe the bug github rate-limit make the plugin unusable
and prone to getting banned by github
Steampipe version (
steampipe -v
) Example: v0.21.2Plugin version (
steampipe plugin list
) Example: v0.39.0Expected behavior github specifies exactly how to handle these status code, we should honor by implementing it.
Additional context