search

spotify / XCRemoteCache

Other
825 stars 50 forks source link

Downloading binaries repeatedly causes 403 rate limit errors with GitHub API #224

Closed Raznic closed 10 months ago

Raznic commented 10 months ago

My integration setup

[x] CocoaPods cocoapods-xcremotecache plugin [ ] Automatic integration using xcprepare integrate ... [ ] Manual integration [ ] Carthage

Expected/desired behavior Recently we started to encounter 403 rate limit exceeded errors while installing/configuring XCRemoteCache in our CI/CD pipelines (see stacktrace below). The error would present itself for a bit across multiple workflows, and then disappear after a period of time. Digging into the source code, the 403 error appears to be coming from the GitHub API for fetching the latest release. From personal experience I know that the GitHub API can be somewhat aggressive in terms of rate limiting requests, so it's probably not surprising this started to happen as we integrated the tool with more and more workflows.

By default our Pods directory is located in the CI/CD job workspace, which gets cleared after each run and thus requires us to redownload the XCRC binaries for each subsequent run. One potential workaround would be to install the XCRC binaries to a more permanent location on our CI/CD workers so they don't have to be downloaded every time. However, it looks like the pre_install hook expects them to always be located in the local user Pod directory. We might be able to do some magic with path traversal, but that feels pretty hacky and prone to error.

### Error

OpenURI::HTTPError - 403 rate limit exceeded
/Users/****/.rbenv/versions/3.1.0/lib/ruby/3.1.0/open-uri.rb:364:in `open_http'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/3.1.0/open-uri.rb:741:in `buffer_open'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/3.1.0/open-uri.rb:212:in `block in open_loop'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/3.1.0/open-uri.rb:210:in `catch'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/3.1.0/open-uri.rb:210:in `open_loop'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/3.1.0/open-uri.rb:151:in `open_uri'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/3.1.0/open-uri.rb:721:in `open'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/3.1.0/open-uri.rb:29:in `open'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-xcremotecache-0.0.18/lib/cocoapods-xcremotecache/command/hooks.rb:287:in `download_latest_xcrc_release'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-xcremotecache-0.0.18/lib/cocoapods-xcremotecache/command/hooks.rb:276:in `download_xcrc_if_needed'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-xcremotecache-0.0.18/lib/cocoapods-xcremotecache/command/hooks.rb:450:in `block in <class:XCRemoteCache>'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/hooks_manager.rb:124:in `block (3 levels) in run'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/user_interface.rb:149:in `message'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/hooks_manager.rb:116:in `block (2 levels) in run'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/hooks_manager.rb:115:in `each'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/hooks_manager.rb:115:in `block in run'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/user_interface.rb:149:in `message'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/hooks_manager.rb:114:in `run'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/installer.rb:675:in `run_plugins_pre_install_hooks'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/installer.rb:229:in `block in prepare'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/user_interface.rb:149:in `message'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/installer.rb:225:in `prepare'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/installer.rb:161:in `install!'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/command/install.rb:52:in `run'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/claide-1.1.0/lib/claide/command.rb:334:in `run'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/lib/cocoapods/command.rb:52:in `run'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/cocoapods-1.12.1/bin/pod:55:in `<top (required)>'
/Users/****/.rbenv/versions/3.1.0/bin/pod:25:in `load'
/Users/****/.rbenv/versions/3.1.0/bin/pod:25:in `<top (required)>'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/cli/exec.rb:58:in `load'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/cli/exec.rb:58:in `kernel_load'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/cli/exec.rb:23:in `run'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/cli.rb:484:in `exec'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/cli.rb:31:in `dispatch'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/cli.rb:25:in `start'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/exe/bundle:48:in `block in <top (required)>'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
/Users/****/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/bundler-2.3.6/exe/bundle:36:in `<top (required)>'
/Users/****/.rbenv/versions/3.1.0/bin/bundle:25:in `load'
/Users/****/.rbenv/versions/3.1.0/bin/bundle:25:in `<main>'

Minimal reproduction of the problem with instructions Run the pre_install hook a bunch of times in a short timespan to trigger the rate-limit on GitHub's API.

Producer Logs N/A - issue is with tool installation

Consumer Logs N/A - issue is with tool installation

Environment

Post build stats N/A - issue is with tool installation

Others N/A

Raznic commented 10 months ago

I suspect this may also be the same issue as #26 based on the description and symptoms.

polac24 commented 10 months ago

Yes, that looks similar and GitHub replied to us that their rate limit should be uncommon, yet didn't provide any details. I guess the avalanche of requests in a very short period of time might indeed hit the rate limit.

If you place the unzipped into XCRC directory, the plugin will not try to redownload it. I can suggest introducing on CI machines a post-checkout and pre-pod_install step to fetch it from your cache (either local or owned-by-you location) to resolve that problem?

Raznic commented 10 months ago

Yup, I think downloading the tools from a cache is a good solution in this case. Thanks for the help!