Regression from v1.0.10: Cannot activate context when using docker-in-docker

[briggySmalls]

Description

Hello again, thanks for this great tool! Can't describe how pleased I am to avoid to CF templates 😅

We use the compose cli in our CI jobs to deploy to ECS (requiring docker in docker). This has been working with v1.0.7 but I'm now looking to upgrade to the latest and greatest. When doing so I cannot seem to activate the ECS context.

Steps to reproduce the issue:

1. Create a docker image that has the Compose CLI and Compose v2 installed:

   echo 'FROM alpine:latest AS downloader
   # Install wget
   RUN apk add wget
   # Download the compose CLI
   RUN wget https://github.com/docker/compose-cli/releases/download/v1.0.17/docker-linux-amd64 && \\
      chmod +x docker-linux-amd64
   # Download compose local
   RUN wget https://github.com/docker/compose-cli/releases/download/v2.0.0-beta.3/docker-compose-linux-amd64 && \\
      chmod 755 docker-compose-linux-amd64
   
   FROM docker:20.10.7
   # Copy the built compose cli
   COPY --from=downloader docker-linux-amd64 /cli/docker
   # Copy compose v2 plugin
   COPY --from=downloader docker-compose-linux-amd64 /root/.docker/cli-plugins/docker-compose
   # Put docker somwhere we expect
   RUN ln -s /usr/local/bin/docker /usr/local/bin/com.docker.cli
   # Add to path
   ENV PATH="/cli:${PATH}"
   # Run
   CMD ["docker"]' > Dockerfile
   docker build -t compose-cli-test .

2. Create a context

   $ docker run -it -e AWS_ACCESS_KEY_ID=abc -e AWS_SECRET_ACCESS_KEY=efg compose-cli-test sh
   # docker context create ecs --from-env ecs

Describe the results you received:

Cannot activate the ecs context:

# docker --context ecs context show
default
# docker context use ecs && docker context show
ecs
default

Therefore executing docker compose up will spin up a local stack.

Describe the results you expected:

To activate an ECS context and be able to deploy to ECS using docker compose up

Note, in order to obtain docker version information I needed to mount the docker engine on my host machine with:

docker run -it -e AWS_ACCESS_KEY_ID=abc -e AWS_SECRET_ACCESS_KEY=def -v /var/run/docker.sock:/var/run/docker.sock compose-cli-test sh

Output of docker version:

Client:
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:51:04 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker context show:
You can also run docker context inspect context-name to give us more details but don't forget to remove sensitive content.

default

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc., 2.0.0-beta.3)

Server:
 Containers: 11
  Running: 1
  Paused: 0
  Stopped: 10
 Images: 229
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 8.748GiB
 Name: docker-desktop
 ID: 6WLX:RSRY:TKNK:2CIA:6ST7:T3OO:ZKJ2:YISU:XOTR:MFPR:36ID:HLFE
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 49
  Goroutines: 59
  System Time: 2021-06-26T11:59:39.6248652Z
  EventsListeners: 3
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS ECS, Azure ACI, local, etc.):

[ndeloof]

Latest compose-cli is still v1.0.17. Compose v2.x is only for local usage, and you don't need it for ECS integration.

[briggySmalls]

Hi @ndeloof! Thanks for looking at this.

Ah OK. Apologies, my mistake. Thanks for clarifying that the Compose v2.x is unnecessary if installing the Compose CLI.

If I update the Dockerfile to skip the installation of Compose v2.x then I still observe the same behaviour. That is, if I update step 1 to that listed below, I cannot activate the ECS context:

1. Create a docker image that has the Compose CLI installed:

   
   echo 'FROM alpine:latest AS downloader
   # Install wget
   RUN apk add wget
   # Download the compose CLI
   RUN wget https://github.com/docker/compose-cli/releases/download/v1.0.17/docker-linux-amd64 && \
   chmod +x docker-linux-amd64

FROM docker:20.10.7

Copy the built compose cli

COPY --from=downloader docker-linux-amd64 /cli/docker

Put docker somwhere we expect

RUN ln -s /usr/local/bin/docker /usr/local/bin/com.docker.cli

Add to path

ENV PATH="/cli:${PATH}"

Run

CMD ["docker"]' > Dockerfile docker build -t compose-cli-test .

## Output of docker version:

Client: Cloud integration: 1.0.17 Version: 20.10.7 API version: 1.41 Go version: go1.13.15 Git commit: f0df350 Built: Wed Jun 2 11:51:04 2021 OS/Arch: linux/amd64 Context: default Experimental: true

Server: Docker Engine - Community Engine: Version: 20.10.7 API version: 1.41 (minimum version 1.12) Go version: go1.13.15 Git commit: b0f5bc3 Built: Wed Jun 2 11:54:58 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.6 GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d runc: Version: 1.0.0-rc95 GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 docker-init: Version: 0.19.0 GitCommit: de40ad0

## Output of docker info

Client: Context: default Debug Mode: false

Server: Containers: 13 Running: 1 Paused: 0 Stopped: 12 Images: 230 Server Version: 20.10.7 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 5.10.25-linuxkit Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 8.748GiB Name: docker-desktop ID: 6WLX:RSRY:TKNK:2CIA:6ST7:T3OO:ZKJ2:YISU:XOTR:MFPR:36ID:HLFE Docker Root Dir: /var/lib/docker Debug Mode: true File Descriptors: 49 Goroutines: 59 System Time: 2021-06-27T04:32:37.1407924Z EventsListeners: 3 HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

[briggySmalls]

After looking at this a bit more, I have identified that the behaviour changes from v1.0.10 onwards.

I simplified my test case to demonstrate the problem:

1. Create the dockerfile (can parameterise the Compose CLI version).

   
   echo 'FROM alpine:latest AS downloader
   # Install wget
   RUN apk add wget
   # Download the compose CLI
   ARG CLI_VERSION
   RUN wget https://github.com/docker/compose-cli/releases/download/${CLI_VERSION}/docker-linux-amd64.tar.gz && \
   tar xzf docker-linux-amd64.tar.gz && \
   chmod +x docker/docker

FROM docker:20.10.7

Copy the built compose cli

COPY --from=downloader docker/docker /cli/docker

Put docker somwhere we expect

RUN ln -s /usr/local/bin/docker /usr/local/bin/com.docker.cli

Add to path

ENV PATH="/cli:${PATH}"

Run

CMD ["docker"]' > Dockerfile

2. Run the test, supplying the desired CLI_VERSION build arg:

docker build --build-arg CLI_VERSION=v1.0.10 -t compose-cli-test . && docker run -it -e AWS_ACCESS_KEY_ID=abc -e AWS_SECRET_ACCESS_KEY=efg -v /var/run/docker.sock:/var/run/docker.sock compose-cli-test sh -c 'docker context create ecs --from-env ecs && docker --context ecs context show'

The test command should print "ecs", the name of the activated context

## Examples with unexpected behaviour

docker build --build-arg CLI_VERSION=v1.0.10 -t compose-cli-test . && docker run -it -e AWS_ACCESS_KEY_ID=abc -e AWS_SECRET_ACCESS_KEY=efg -v /var/run/docker.sock:/var/run/docker.sock compose-cli-test sh -c 'docker context create ecs --from-env ecs && docker --context ecs context show' [+] Building 0.7s (11/11) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 37B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/docker:20.10.7 0.5s => [internal] load metadata for docker.io/library/alpine:latest 0.0s => [stage-1 1/3] FROM docker.io/library/docker:20.10.7@sha256:bfc499cef26daa22da31b76be1752813a6921ee1fa1dd 0.0s => [downloader 1/3] FROM docker.io/library/alpine:latest 0.0s => CACHED [downloader 2/3] RUN apk add wget 0.0s => CACHED [downloader 3/3] RUN wget https://github.com/docker/compose-cli/releases/download/v1.0.10/docker- 0.0s => CACHED [stage-1 2/3] COPY --from=downloader docker/docker /cli/docker 0.0s => CACHED [stage-1 3/3] RUN ln -s /usr/local/bin/docker /usr/local/bin/com.docker.cli 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:ccd580b42fde16b3806f4f4f637e22d3a3b1fb5a5fe4ffdfa4907981802e799d 0.0s => => naming to docker.io/library/compose-cli-test 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them Successfully created ecs context "ecs" default

## Examples where behaviour is as expected

Any release prior to 1.0.10:

docker build --build-arg CLI_VERSION=v1.0.9 -t compose-cli-test . && docker run -it -e AWS_ACCESS_KEY_ID=abc -e AWS_SECRET_ACCESS_KEY=efg -v /var/run/docker.sock:/var/run/docker.sock compose-cli-test sh -c 'docker context create ecs --from-env ecs && docker --context ecs context show' [+] Building 0.8s (11/11) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 37B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/docker:20.10.7 0.6s => [internal] load metadata for docker.io/library/alpine:latest 0.0s => [stage-1 1/3] FROM docker.io/library/docker:20.10.7@sha256:bfc499cef26daa22da31b76be1752813a6921ee1fa1dd 0.0s => [downloader 1/3] FROM docker.io/library/alpine:latest 0.0s => CACHED [downloader 2/3] RUN apk add wget 0.0s => CACHED [downloader 3/3] RUN wget https://github.com/docker/compose-cli/releases/download/v1.0.9/docker-l 0.0s => CACHED [stage-1 2/3] COPY --from=downloader docker/docker /cli/docker 0.0s => CACHED [stage-1 3/3] RUN ln -s /usr/local/bin/docker /usr/local/bin/com.docker.cli 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:566f9cad212babb19efe3be74609ef010ba765b438c584628e802e51cc6c26ad 0.0s => => naming to docker.io/library/compose-cli-test 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them Successfully created ecs context "ecs" ecs

## Other thoughts

Release v1.0.17 (the latest) cannot be tested like this because that release _doesn't_ supply a tar, instead it supplies the tool as an executable. My previous comment demonstrates the behaviour exists there too.

[briggySmalls]

I'd suggest that this is not in fact a ~question but a potential ~bug?

[briggySmalls]

@ndeloof please could you review my updates? I do believe this to be a bug. 🙏

[tmeijn]

Hey @briggySmalls this was also something I was struggling with, but I managed to get it to work with a workaround. Take a look at my project and especially here, where I'm using socat to reroute the docker.sock to the DinD service. Hope this helps you!

[sam-briggs-depop]

Nice one!

[bcouetil]

Thanks @tmeijn :pray: but this is still a workaround for some piece of compose-cli using /var/run/docker.sock instead of DOCKER_HOST, right ?

[bcouetil]

This did not work for me, I'm also on Gitlab but without TLS... I tried this with no luck :

socat -d -d -d -lf socat.log UNIX-LISTEN:/var/run/docker.sock,reuseaddr,unlink-early,fork TCP:docker:2375

No error but I cannot switch context.

I'm OK with switching to TLS but I don't know how to handle the CERT part :sweat:

[tmeijn]

Thanks @tmeijn pray but this is still a workaround for some piece of compose-cli using /var/run/docker.sock instead of DOCKER_HOST, right ?

Hi @bcouetil Indeed, I'll clarify in my comment that it's indeed a workaround!

It's been some time since I looked into this but I THINK this is the relevant code which resets the context to default when DOCKER_HOST is present: https://github.com/docker/compose-cli/blob/fa05d4397ad4c3323fa5d15fef14355e1833adee/cli/config/flags.go#L58;60 (@ndeloof is this true? And if so, could this be changed? Take a look at my workaround repo, seems to work fine for ECS context, but I'm not aware of other considerations)

This did not work for me, I'm also on Gitlab but without TLS... I tried this with no luck : So I assume it's self-hosted?

No error but I cannot switch context.

socat.log also has no data inside?

I'm OK with switching to TLS but I don't know how to handle the CERT part

This article might help you: https://about.gitlab.com/blog/2019/07/31/docker-in-docker-with-docker-19-dot-03/

[bcouetil]

Thank you for your inputs.

I'm on Gitlab public runners. build/push works great without TLS. But I can't sort things out with TLS. The article is for Docker 19, and I'm trying on Docker 20 with no luck (or should I try on v19 ?).

For now, I deployed successfully using compose-cli v1.0.9, if anyone want the install command :

curl -LO https://github.com/docker/compose-cli/releases/download/v1.0.9/docker-linux-amd64.tar.gz && tar -xf docker-linux-amd64.tar.gz && chmod +x docker/docker && mv docker/docker /usr/local/bin/docker && ln -s /usr/bin/docker /usr/local/bin/com.docker.cli

[tmeijn]

@bcouetil I'm also on GitLab.com Shared Runners. Please examine the .gitlab-ci.yml, ecs-review.sh and Dockerfile in my repo. If you copy those over and start from there I see no reason it shouldn't also work for you.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.