GitHub Rate Limiting on download

[ghost]

Hello,

We are intermittently running into GitHub rate limiting when downloading tini in our docker builds.

Dockerfile

from node:12
...
# Add Tini
ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini # <-- Fails due to rate limiting
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]
...

Per https://developer.github.com/v3/#increasing-the-unauthenticated-rate-limit-for-oauth-applications and other GitHub scripts we have we could authenticate or use an Authorization header but it does not look like ADD/COPY support that: https://docs.docker.com/engine/reference/builder/#add

Is it possible to get tini from apt/apt-get, or would we need to use something like curl to authenticate and get the tini binary?

[krallin]

Currently, downloading from Github is the only way to get a pre-built Tini binary (but there are instructions for building from source in the README if you want that). The rate limit documentation you're sharing is for API access though — I don't think this should apply to releases downloads.

Out of curiosity, where are you running this from and can you share the full HTTP response you get from Github? This is the first I hear of rate limiting on Github releases and I wonder if this could be more about your IP in particular sending too many downloads than about Tini receiving too many downloads in general.

[ghost]

I think you're correct on the API docs I linked. Not sure where the Authorization: token GITHUB_API_TOKEN header is documented.

The tini download runs in a Docker build (docker build ...) In Travis-CI. I believe you're right about the IP rate limiting for releases. Below is an example of another release we download where we use the GITHUB_API_TOKEN to increse the amount of downloads we can do in our CI environment:

get-release.sh

#!/usr/bin/env bash
set -e

...

VERSION=${VERSION:-v1.2.3}

echo "#########################################"
echo "# Installing ${VERSION} #"
echo "#########################################"

RELEASES_URL=https://api.github.com/repos/org/repo/releases/tags/${VERSION}
RELEASE_RESULTS=`curl -sSL -H "Authorization: token ${GITHUB_API_TOKEN}" ${RELEASES_URL}`
DOWNLOAD_URL=`echo ${RELEASE_RESULTS} | jq -r '.assets[] | select(.name | contains("linux")).browser_download_url'`
curl -sSL ${DOWNLOAD_URL} | sudo tar -xzC /usr/local/bin

I've been able to reproduce the rate limiting locally (Mac/OSX) using the below script and curl.

spam-tini-download.sh

#!/usr/bin/env bash

set -e

echo "-------------------------------------------------"
echo "| Spamming tini download 1000 times"
echo "-------------------------------------------------"

for i in {1..1000}
do
   curl -LsS -o /dev/null -w "%{http_code}" https://github.com/krallin/tini/releases/download/v0.18.0/tini && echo " #${i}" & # 
done

echo "-------------------------------------------------"
echo "| Tasks spawned. Waiting 10 seconds..."
echo "-------------------------------------------------"

sleep 10s

echo "-------------------------------------------------"
echo "| Rate limited tini download"
echo "-------------------------------------------------"

curl -ILsS https://github.com/krallin/tini/releases/download/v0.18.0/tini

echo "-------------------------------------------------"
echo "| Fixed with Github API token"
echo "-------------------------------------------------"

curl -ILsS -H "Authorization: token YOUR_GITHUB_API_TOKEN" https://github.com/krallin/tini/releases/download/v0.18.0/tini

echo "-------------------------------------------------"
echo "| Done"
echo "-------------------------------------------------"

Output

$ ./spam-tini-download.sh                                                                                                                                                                                     

-------------------------------------------------
| Spamming tini download 1000 times
-------------------------------------------------
200 #5

...

-------------------------------------------------
| Tasks spawned. Waiting 10 seconds...
-------------------------------------------------

...

429 #555
-------------------------------------------------
| Rate limited tini download
-------------------------------------------------
HTTP/1.1 429 too many requests
Date: Tue, 21 Apr 2020 21:10:45 GMT
Server: Varnish
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'
Content-Type: text/html; charset=utf-8
Retry-After: 60
Content-Length: 1474
X-GitHub-Request-Id: REDACTED

-------------------------------------------------
| Fixed with Github API token
-------------------------------------------------
HTTP/1.1 302 Found
Server: GitHub.com
Date: Tue, 21 Apr 2020 21:10:45 GMT
Content-Type: text/html; charset=utf-8
Status: 302 Found
Vary: X-PJAX
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/31432573/461babf8-4548-11e8-836f-ad8740b7b1fa?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=REDACTED%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200421T211045Z&X-Amz-Expires=300&X-Amz-Signature=REDACTED&X-Amz-SignedHeaders=host&actor_id=0&repo_id=31432573&response-content-disposition=attachment%3B%20filename%3Dtini&response-content-type=application%2Foctet-stream
Cache-Control: no-cache
Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Wed, 21 Apr 2021 21:10:45 GMT; secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com
Vary: Accept-Encoding, Accept, X-Requested-With
X-GitHub-Request-Id: REDACTED

HTTP/1.1 403 Forbidden
x-amz-request-id: REDACTED
x-amz-id-2: REDACTED
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Tue, 21 Apr 2020 21:10:45 GMT
Server: AmazonS3

-------------------------------------------------
| Done
-------------------------------------------------

Of note is the 403 on the S3 redirect. Navgating in a browser there (https://github.com/krallin/tini/releases/download/v0.18.0/tini) worked just fine. Likely a red herring.

As mentioned in the issue description, from what I know of ADD we can't add addtional headers. We could also switch to curl. Just prefer not to have to pass that GITHUB_API_TOKEN into or Docker build process as a build arg.

[krallin]

I think using Curl here to set the token would your best shot here if you need it, yes. Since I haven't seen further reports of this, I'll close the issue.