search

docker / compose

Define and run multi-container applications with Docker
https://docs.docker.com/compose/
Apache License 2.0
33.68k stars 5.19k forks source link

[BUG] docker-compose 2.20.3 rewriting environment variables #10902

Closed grahamb closed 1 year ago

grahamb commented 1 year ago

Description

This sounds very strange, but it appears that docker-compose v2.20.3 is rewriting environment variables, specifically http_proxy and https_proxy (and their upper-case variants).

I have http(s)_proxy and no_proxy variables set in my shell, in ~/.docker/config.json, and in /etc/systemd/system/docker.service.d/http-proxy.conf

My shell variables:

❯ env | grep -i proxy                                          
https_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
http_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
HTTPS_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
HTTP_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
NO_PROXY=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,sentry.its.sfu.ca,twilio.com
no_proxy=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,sentry.its.sfu.ca,twilio.com

I have a very simple test Dockerfile:

FROM alpine:latest
RUN env | grep -i proxy

If I build the image with Docker (not docker-compose), the proxy variables output by the build are what I would expect:

❯ docker buildx build --tag local/dev:latest --progress plain --no-cache .
#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context: 210B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.05kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.3s

#4 [1/2] FROM docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
#4 CACHED

#5 [2/2] RUN env | grep -i proxy
#5 0.322 HTTPS_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.322 no_proxy=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 0.322 NO_PROXY=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 0.322 https_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.322 http_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.322 HTTP_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 DONE 0.4s

#6 exporting to image
#6 exporting layers 0.0s done
#6 writing image sha256:c3fb9a84471a3c7ab852233e9dedb028f5114813d610ce20c3939d891147f6e0 done
#6 naming to docker.io/local/dev:latest done
#6 DONE 0.0s

However, if I do a docker-compose build, the variables get re-written:

❯ docker-compose --progress plain build web --no-cache 
#0 building with "default" instance using docker driver

#1 [web internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.05kB done
#1 DONE 0.0s

#2 [web internal] load .dockerignore
#2 transferring context: 210B done
#2 DONE 0.0s

#3 [web internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.3s

#4 [web 1/2] FROM docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
#4 CACHED

#5 [web 2/2] RUN env | grep -i proxy
#5 0.265 HTTPS_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.265 no_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.265 NO_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.265 https_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.265 http_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.265 HTTP_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 DONE 0.3s

#6 [web] exporting to image
#6 exporting layers 0.0s done
#6 writing image sha256:aa22651d984e621e4a30f1b41692d7ef013fbf0470cad821b238fb71457ab127 done
#6 naming to docker.io/library/grouperui-lite-web done
#6 DONE 0.0s

Note that my no_proxy and NO_PROXY variables have been changed and match the https_proxy et al variables.

I downgraded to the previous docker-compose release (2.20.2), and the docker-compose build command returns the expected output.

❯ /tmp/docker-compose_2.20.2 --version && /tmp/docker-compose_2.20.2 --progress plain build web --no-cache 
Docker Compose version v2.20.2
#1 [web internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.05kB done
#1 DONE 0.0s

#2 [web internal] load .dockerignore
#2 transferring context: 210B done
#2 DONE 0.0s

#3 [web auth] library/alpine:pull token for registry-1.docker.io
#3 DONE 0.0s

#4 [web internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.6s

#5 [web 1/2] FROM docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
#5 CACHED

#6 [web 2/2] RUN env | grep -i proxy
#6 0.244 HTTPS_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
#6 0.244 no_proxy=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#6 0.244 NO_PROXY=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#6 0.244 https_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
#6 0.244 http_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
#6 0.244 HTTP_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
#6 DONE 0.3s

#7 [web] exporting to image
#7 exporting layers 0.0s done
#7 writing image sha256:9a94e0513585f50ea6252d8489610233a4342349112b3177ba334240b98a9f52 done
#7 naming to docker.io/library/grouperui-lite-web done
#7 DONE 0.0s

Note that no_proxy and NO_PROXY match what is set in my shell.

With the alpine:latest image, it's changing no_proxy and NO_PROXY to the values of https_proxy. However, in other images (e.g. ubuntu:latest, and some other custom-built images) it's the opposite; https_proxy (and the uppercase variants) are set to the value of no_proxy. For example, with ubuntu:latest:

❯ docker-compose --progress plain build web --no-cache                                                    
#0 building with "default" instance using docker driver

#1 [web internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.05kB done
#1 DONE 0.0s

#2 [web internal] load .dockerignore
#2 transferring context: 210B done
#2 DONE 0.0s

#3 [web internal] load metadata for docker.io/library/ubuntu:latest
#3 DONE 0.3s

#4 [web 1/2] FROM docker.io/library/ubuntu:latest@sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508
#4 CACHED

#5 [web 2/2] RUN env | grep -i proxy
#5 0.260 HTTPS_PROXY=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 0.260 no_proxy=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 0.260 NO_PROXY=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 0.260 https_proxy=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 0.260 http_proxy=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 0.260 HTTP_PROXY=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 DONE 0.3s

#6 [web] exporting to image
#6 exporting layers done
#6 writing image sha256:806c020b53b0b0ec6cb766b8abdf2720f0717d2af2b8c46c963b529857365b18 done
#6 naming to docker.io/library/grouperui-lite-web done
#6 DONE 0.0s

Steps To Reproduce

  1. Have http(s)_proxy and no_proxy environment variables set in your shell
  2. Have a Dockerfile as outlined above
  3. Run docker-compose --progress plain build --no-cache

Expected behaviour: environment variables output during the build should match those as set in the shell Actual behaviour: docker-compose is rewriting the variables' values

Compose Version

Docker Compose version v2.20.2

Docker Environment

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 8
  Running: 8
  Paused: 0
  Stopped: 0
 Images: 41
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.18.0-477.21.1.el8_8.x86_64
 Operating System: Red Hat Enterprise Linux 8.8 (Ootpa)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 11.42GiB
 Name: lcp-grahamb-dev.dc.sfu.ca
 ID: ad8e5c73-6f60-49f1-8b5a-447568a1f768
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http://bby-vcontrol-proxy.its.sfu.ca:8080
 HTTPS Proxy: http://bby-vcontrol-proxy.its.sfu.ca:8080
 No Proxy: *.sfu.ca
 Username: grahamb
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

grahamb commented 1 year ago

This comment in pkg/compose/build.go is suspect:

// Finally, standard proxy variables based on the Docker client configuration are added, but will not overwrite
// any values if already present.
ndeloof commented 1 year ago

Do you have some proxy configuration declared in .docker/config.json ?

grahamb commented 1 year ago

Yes, I do:

{
    "proxies": {
        "default": {
            "httpProxy": "http://bby-vcontrol-proxy.its.sfu.ca:8080",
            "httpsProxy": "http://bby-vcontrol-proxy.its.sfu.ca:8080",
            "noProxy": "localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com"
        }
    }
}
ndeloof commented 1 year ago

ok, do you have the same behavior running docker buildx build . with your service's Dockerfile ?

grahamb commented 1 year ago

No - I mentioned that in the opening post, but it's admittedly kind of buried in there.

If I build the image with Docker (not docker-compose), the proxy variables output by the build are what I would expect:

❯ docker buildx build --tag local/dev:latest --progress plain --no-cache .
#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context: 210B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.05kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.3s

#4 [1/2] FROM docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
#4 CACHED

#5 [2/2] RUN env | grep -i proxy
#5 0.322 HTTPS_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.322 no_proxy=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 0.322 NO_PROXY=localhost,127.0.0.1,10.28.130.135,sfu.ca,www.sfu.ca,*.sfu.ca,*.its.sfu.ca,.sfu.ca,.its.sfu.ca,its.sfu.ca,github.sfu.ca,lookups.twilio.com,api.twillio.com
#5 0.322 https_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.322 http_proxy=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 0.322 HTTP_PROXY=http://bby-vcontrol-proxy.its.sfu.ca:8080
#5 DONE 0.4s

#6 exporting to image
#6 exporting layers 0.0s done
#6 writing image sha256:c3fb9a84471a3c7ab852233e9dedb028f5114813d610ce20c3939d891147f6e0 done
#6 naming to docker.io/local/dev:latest done
#6 DONE 0.0s
ndeloof commented 1 year ago

the root cause is a variable collision iterating proxy config in .docker/config.json, a very common Go issue which strikes back on a regular basis