[BUG] docker compose cp doesn't work for services executed via docker compose run

[synek317]

Description

The very useful docker compose cp command does not work when I use docker compose run {service}.

Steps To Reproduce

Example docker-compose.yml:

services:
  foo:
    image: alpine

What works:

docker compose up
docker compose cp foo:/etc/alpine-release ./
docker compose down

What doesn't work:

docker compose run foo echo
docker compose cp foo:/etc/alpine-release ./

Error:

no container found for service "foo"

Expectation: It works the same with docker compose up or at least there is another way to copy the file.

Compose Version

Docker Compose version v2.15.1 (both, docker compose version and docker-compose version)

Docker Environment

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.3)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.18)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.6.0)

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 54
 Server Version: 20.10.23
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 11.68GiB
 Name: docker-desktop
 ID: YCB4:AIWC:GZ2L:IWPO:MGNR:4JOP:AUQC:KNJQ:KPOR:CMD6:GUIJ:ETLS
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[kamilpi]

Hi @synek317

I would like to say "It's not a bug but a feature" and if you will explore the approach and differences between run and up you can see what is going on inside.

In short words:

The run command collects your service definition but in the end, it is just a docker run command prepared by your service definition.

I give you some insight into this case. Here is a part of the code when docker compose service definition is collecting https://github.com/docker/compose/blob/v2/cmd/compose/run.go#L219 at the very end do the docker run the command without attaching to the service: https://github.com/docker/compose/blob/v2/pkg/compose/run.go#L41. This is one-off container created.

This one-off container option has one rule. As you may probably know in the runtime process docker creates additional labels to be able to manage services. Based on this I will just show you different labels:

run:

"com.docker.compose.oneoff": "True"

up:

"com.docker.compose.oneoff": "False"

Compose uses also this label to list services: https://github.com/docker/compose/blob/v2/pkg/compose/ps.go#L32

Will preempt your next question: You are not able to override this label.

In the end, I have a solution for you, only the command, without explanation. Check this out:

docker-compose run --name foo foo
docker cp foo:/etc/alpine-release ./

[ndeloof]

Could you please explain what use-case would make this feature useful ?

As a workaround, you obviously can use docker cp .. for one-off containers (container started by docker compose run ...).

[ndeloof]

agree in theory, but I noticed we let you docker compose exec into a one off container if no service is running for target service, so would like to better understand the use-case for this feature request

[synek317]

@ndeloof

My use case is test development for CLI application that downloads files from various online services (webservices, ftps etc.). I have a docker-compose.yml with multiple services. One of them (the one that I run) is my CLI app, others are the aforementioned external services.

docker compose run has many advantages: it ends once my app ends, it allows me to pass arguments to my application. Initially, I just used bind volume but I quickly encountered an issue with file permissions on Linux. Basically, files created by the docker container on the host machine were owned by root:root what breaks further processing.

This is how I discovered docker compose cp. It doesn't have this issue but it doesn't work with docker compose run. So I had to develop another workaround, which is docker compose up --abort-on-container-exit --exit-code-from. Which then needed yet another workaround to pass cmd line arguments to my service (I use env variable for that).

I think that you can see how many times did I use the word workaround. It's definitely not perfect.

And in the end, it is counter-intuitive that docker compose cp works with some containers created with docker compose but not with others. It becomes clearer when you know the underlying details that @kamilpi explained. Still, I think being intuitive even for newcomers makes tools better.

[iamthad]

I, too, was confused that I can docker compose exec in containers created by docker compose run but I cannot docker compose cp to them.

[ndeloof]

the main reason we have this limitation is that docker compose cp targets multiple containers (all replicas for service), while docker compose exec only target one