search

advanced-security / gh-sbom

Generate SBOMs with gh CLI
MIT License
164 stars 13 forks source link

encountering secondary rate limit restrictions #9

Closed travi closed 1 year ago

travi commented 1 year ago

when executing against projects where the dependency graph is tracking several pages of dependencies, we are encountering secondary rate limits before the full query result can be processed. is there a way to configure the client to honor the retry-after/x-ratelimit-reset headers?

for completeness, this is the error we are seeing in this case:

non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"

in addition, we are sometimes seeing a timeout error, before encountering the secondary rate limit. Is this a known issue?

Message: timedout, Locations: [{Line:1 Column:155}]
steiza commented 1 year ago

Sorry for the delay! As you discovered, it can take many API calls to generate a SBOM for a large repository, or fail altogether for very large repositories.

The Dependency Graph team was kind enough to implement a server-side SBOM generator for SPDX, which is a single API call and much, much faster. The gh-sbom v0.0.9 release makes use of this feature - give it a try and let us know if that works for you?

You'll need to update gh-sbom with:

$ gh ext remove advanced-security/gh-sbom
$ gh ext install advanced-security/gh-sbom
travi commented 1 year ago

Thanks for the update. Great to hear that the API generation is available. Will test it out soon!

travi commented 1 year ago

this is great and so fast :)

one thing to note, the README still mentions that GHES 3.8 is needed. when using against GHES, the new rest endpoint for the sbom doesnt yet exist in v3.8. since https://github.com/github/roadmap/issues/626 highlights that these features are to be expected in v3.9, what would be the appropriate update to make that clear in the README until that is released?

steiza commented 1 year ago

Great call-out! I have done so with https://github.com/advanced-security/gh-sbom/commit/5e0c9242b0b7dcfbdf19cd111a4335a3193a0054.