search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
757 stars 86 forks source link

Dockerfile ADD command with remote URL on GitHub in a specific AWS region #1095

Open mtilson opened 4 years ago

mtilson commented 4 years ago

There is a strange issue with building docker images from Dockerfile with command ADD <src> (when <src> is remote file URL pointing to GitHub only) which appears on AWS EC2 instances in specific region only (us-east-2) and related to the latest docker-ce versions only (19.03.9 and above). The below examples are provided for Ubuntu 18.04, but the issue is applied to Ubuntu 20.04 too.

Expected behavior

Actual behavior

Steps to reproduce the behavior

We did the following preparation steps on every tested environments (see Additional environment details (AWS, VirtualBox, physical, etc.) -> Clouds section below for the tested environment instances)

$ sudo -i
# apt-get update
# apt-get upgrade
# apt-get install apt-transport-https ca-certificates curl software-properties-common
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg |  apt-key add -
# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
# apt-get update
# apt-get install docker-ce=5:19.03.8~3-0~ubuntu-bionic docker-ce-cli=5:19.03.8~3-0~ubuntu-bionic
# mkdir /root/.tmp && cd /root/.tmp
# cat > Dockerfile << EOF
FROM alpine:latest
ADD "https://github.com/docker/docker-ce/archive/v19.03.12.tar.gz" /docker-ce/
EOF

Then we did the tests with docker-ce 19.03.8

Then we did the tests with docker-ce 19.03.9 (and newer ones)

FROM alpine:latest
ADD "https://www.kernel.org/pub/software/scm/git/git-2.28.0.tar.gz" /git-src/
ADD "http://dl-cdn.alpinelinux.org/alpine/v3.12/releases/x86_64/alpine-minirootfs-3.12.0-x86_64.tar.gz" /alpine-mrfs/

Output of docker version (GCP):

# docker version
Client: Docker Engine - Community
 Version:           19.03.9
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        9d988398e7
 Built:             Fri May 15 00:25:18 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.9
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       9d988398e7
  Built:            Fri May 15 00:23:50 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info (GCP):

# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 19.03.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1021-gcp
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 3.597GiB
 Name: instance-1
 ID: FC4D:ABZK:I6UL:FJMW:GGEN:527T:OZOA:SMG7:ODLH:V4TA:E7KU:4LEZ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Output of docker version (AWS):

# docker version
Client: Docker Engine - Community
 Version:           19.03.9
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        9d988398e7
 Built:             Fri May 15 00:25:18 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.9
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       9d988398e7
  Built:            Fri May 15 00:23:50 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info (AWS):

# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 19.03.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.3.0-1032-aws
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 978.6MiB
 Name: ip-172-31-11-238
 ID: R3KL:EBR6:CV6K:YK3H:CJK4:FQJC:EJFU:SI5T:FTNB:NTO7:QO6A:DHUB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

thaJeztah commented 4 years ago

Thanks for your detailed report!

Interesting. So looking at the diff between 19.03.8 and 19.03.9; https://github.com/moby/moby/compare/v19.03.8...v19.03.9?w=1

Potential suspects could be;

Although I don't think DNS would play a role here (famous last words). I wonder what's the reason the request is blocked by GitHub. I do recall we had an incident recently where changes a combination of changes in Amazon Linux's configuration, and updated botnet-protection rules caused traffic to be blocked https://www.docker.com/blog/docker-hub-incident-review-5-july-2020/ by Docker Hub's CDN. Wondering if (e.g.) requests made with Go 1.13+ differ and cause some detection to be triggered. It's curious that it only affects us-east-2 though 🤔

Have you tried contacting GitHub support if they could see why requests were blocked?

thaJeztah commented 4 years ago

Are you seeing the same problem if you use BuildKit for building? You can enable buildkit by setting the DOCKER_BUILDKIT=1 environment variable on the machine where the client (docker cli) runs (DOCKER_BUILDKIT=1 docker build .....)

thaJeztah commented 4 years ago

Just a very rudimentary test to check if there's a differences in the request made by Docker 19.03.8 and 19.03.9 when ADD-ing a file during build (not on AWS, so not taking that into account);

With 19.03.8

GET /index.html HTTP/1.1
Host: <IP-address>:9100
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

With 19.03.9

GET /index.html HTTP/1.1
Host: <IP-address>:9100
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

That said, the github downloads would be using TLS, and a redirect

```console curl -vvv https://github.com/docker/docker-ce/archive/v19.03.12.tar.gz * Trying 140.82.121.4:443... * Connected to github.com (140.82.121.4) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com * start date: May 5 00:00:00 2020 GMT * expire date: May 10 12:00:00 2022 GMT * subjectAltName: host "github.com" matched cert's "github.com" * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA * SSL certificate verify ok. > GET /docker/docker-ce/archive/v19.03.12.tar.gz HTTP/1.1 > Host: github.com > User-Agent: curl/7.69.1 > Accept: */* > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * old SSL session ID is stale, removing * Mark bundle as not supporting multiuse < HTTP/1.1 302 Found < date: Mon, 31 Aug 2020 11:47:14 GMT < content-type: text/html; charset=utf-8 < server: GitHub.com < status: 302 Found < vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding < location: https://codeload.github.com/docker/docker-ce/tar.gz/v19.03.12 < cache-control: max-age=0, private < strict-transport-security: max-age=31536000; includeSubdomains; preload < x-frame-options: deny < x-content-type-options: nosniff < x-xss-protection: 1; mode=block < expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors" < content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events wss://alive.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/socket-worker.js gist.github.com/socket-worker.js < Content-Length: 127 < X-GitHub-Request-Id: 86B4:DB63:A1E85F0:F0EF5E9:5F4CE341 < * Connection #0 to host github.com left intact ```
mtilson commented 4 years ago

Are you seeing the same problem if you use BuildKit for building? You can enable buildkit by setting the DOCKER_BUILDKIT=1 environment variable on the machine where the client (docker cli) runs (DOCKER_BUILDKIT=1 docker build .....)

# DOCKER_BUILDKIT=1 docker build -t test-add-command:19.03.9 .
[+] Building 0.6s (5/6)
 => [internal] load build definition from Dockerfile                                                                                                    0.0s
 => => transferring dockerfile: 135B                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                         0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                        0.4s
 => ERROR https://github.com/docker/docker-ce/archive/v19.03.12.tar.gz                                                                                  0.2s
 => CANCELED [1/2] FROM docker.io/library/alpine:latest@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321                         0.2s
 => => resolve docker.io/library/alpine:latest@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321                                  0.0s
 => => sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 1.64kB / 1.64kB                                                          0.0s
 => => sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 528B / 528B                                                              0.0s
 => => sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e 1.51kB / 1.51kB                                                          0.0s
 => => sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 2.80MB / 2.80MB                                                          0.2s
 => => extracting sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c                                                               0.0s
------
 > https://github.com/docker/docker-ce/archive/v19.03.12.tar.gz:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: invalid response status 403
mtilson commented 4 years ago

Have you tried contacting GitHub support if they could see why requests were blocked?

Here it goes: https://github.community/t/dockerfile-add-command-with-remote-url-on-github-in-a-specific-aws-region/130432

PS: Seems like there is one more involved party ... AWS ))