search

jekyll / github-metadata

Jekyll plugin to propagate the `site.github` namespace and set default values for use with GitHub Pages.
https://jekyll.github.io/github-metadata/
MIT License
278 stars 126 forks source link

Limit and/or Cache API requests to avoid: 403 - API rate limit exceeded for installation ID ___ #266

Open trinitronx opened 10 months ago

trinitronx commented 10 months ago

I just switched to using Jekyll 4.3.2 and tried to deploy a static site using this theme, which lists all current GitHub repositories on the generated static site page. To do this, it uses jekyll-github-metadata gem (version 2.16.0) to gather all repositories for my GitHub user.

Apparently, this results in a lot of API requests (for my user, over ~486!) and quickly hits the rate limit when using GITHUB_TOKEN in a GitHub Actions workflow.

Would it be possible to cache or limit API requests further so static site deployments will complete successfully in GitHub Actions?

For example: The jekyll build --verbose output shows each repo is queried for both contributors and releases, but the template doesn't even need this information

```console Logging at level: debug Jekyll Version: 4.3.2 Configuration file: /mnt/terabyte/src/sites/trinitronx.github.io/_config.yml Configuration file: /mnt/terabyte/src/sites/trinitronx.github.io/_config_dev.yml Requiring: jekyll-github-metadata To use retry middleware with Faraday v2.0+, install `faraday-retry` gem GitHub Metadata: Initializing... Source: /mnt/terabyte/src/sites/trinitronx.github.io Destination: /mnt/terabyte/src/sites/trinitronx.github.io/_site Incremental build: enabled Generating... Reading: /_layouts/default.html EntryFilter: excluded /LICENSE.neumorphism EntryFilter: excluded /README.md EntryFilter: excluded /docs EntryFilter: excluded /package.json EntryFilter: excluded /node_modules EntryFilter: excluded /.jekyll-cache EntryFilter: excluded /.github EntryFilter: excluded /.ruby-version EntryFilter: excluded /Gemfile EntryFilter: excluded /yarn.lock EntryFilter: excluded /.ruby-gemset EntryFilter: excluded /Gemfile.lock EntryFilter: excluded /.gitignore EntryFilter: excluded /gulpfile.js Reading: index.html Reading: _data/projects.yml Reading: _data/skills-frameworks.yml Reading: _data/skills-languages.yml Reading: _data/skills-tools.yml Reading: _data/timeline.yml Rendering: index.html Pre-Render Hooks: index.html Rendering Markup: index.html Post-Convert Hooks: index.html Rendering Layout: index.html GitHub Metadata: Generating for trinitronx/trinitronx.github.io GitHub Metadata: Calling @client.list_repos("trinitronx", {:type=>"public", :accept=>"application/vnd.github.mercy-preview+json"}) GitHub Metadata: Calling @client.releases("trinitronx/action-doctl") GitHub Metadata: Calling @client.contributors("trinitronx/action-doctl") GitHub Metadata: Calling @client.releases("trinitronx/ansible") GitHub Metadata: Calling @client.contributors("trinitronx/ansible") GitHub Metadata: Calling @client.releases("trinitronx/ansible-examples") GitHub Metadata: Calling @client.contributors("trinitronx/ansible-examples") [... SNIP ...] [... TOTAL number of releases + contributors API log lines = 486 ...] [... SNIP ...] GitHub Metadata: Calling @client.releases("trinitronx/zengge-lightcontrol") GitHub Metadata: Calling @client.contributors("trinitronx/zengge-lightcontrol") Writing: /mnt/terabyte/src/sites/trinitronx.github.io/_site/index.html Writing Metadata: .jekyll-metadata done in 177.699 seconds. Auto-regeneration: disabled. Use --watch to enable. [18:31:02] Finished 'jekyll' after 2.97 min ```

trinitronx commented 10 months ago

Some ideas:

In case it helps identify the code paths involved for the API calls...

Expand for stacktrace when pressing Ctrl+C during the API call loop

```console /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1018:in `initialize': Interrupt from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1018:in `open' from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1018:in `block in connect' from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout' from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/timeout.rb:117:in `timeout' from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1016:in `connect' from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:995:in `do_start' from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:984:in `start' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:112:in `request_with_wrapped_block' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:102:in `perform_request' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:66:in `block in call' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/adapter.rb:45:in `connection' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:65:in `call' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/middleware.rb:17:in `call' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/middleware.rb:17:in `call' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/middleware.rb:17:in `call' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/middleware/follow_redirects.rb:73:in `perform_with_redirection' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/middleware/follow_redirects.rb:61:in `call' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/rack_builder.rb:153:in `build_response' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/connection.rb:444:in `run_request' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/connection.rb:200:in `get' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/sawyer-0.9.2/lib/sawyer/agent.rb:99:in `call' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/connection.rb:156:in `request' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/connection.rb:84:in `paginate' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/client/releases.rb:15:in `releases' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:63:in `public_send' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:63:in `block in method_missing' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:92:in `save_from_errors' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:63:in `method_missing' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/repository.rb:123:in `block (3 levels) in owner_public_repositories' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/value.rb:53:in `call_or_value' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/value.rb:29:in `render' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:35:in `sanitize' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:55:in `block in sanitize_resource' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:54:in `each' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:54:in `each_with_object' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:54:in `sanitize_resource' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:38:in `sanitize' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:23:in `block in sanitize' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:23:in `map' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:23:in `sanitize' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/value.rb:29:in `render' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/repository.rb:216:in `memoize_value' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/repository.rb:120:in `owner_public_repositories' from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/forwardable.rb:238:in `public_repositories' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-4.3.2/lib/jekyll/drops/drop.rb:128:in `public_send' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-4.3.2/lib/jekyll/drops/drop.rb:128:in `[]' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/liquid-4.0.4/lib/liquid/context.rb:197:in `lookup_and_evaluate' from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/liquid-4.0.4/lib/liquid/variable_lookup.rb:48:in `block in evaluate' ```

aspiers commented 5 months ago

I see something similar; it's querying releases and contributors for ALL repos I own, even though I'm only using Jekyll on a single one - why is it doing this?!