docker-compose copy file or directory to container

[ghost]

we miss a possibility to copy a file or directory using docker-compose. I find this really useful. Please check many +1 in premature closed https://github.com/docker/compose/issues/2105

[ndeloof]

@robclancy the anti-pattern is the way some might use this command. Could say the same about docker run --privileged

We introduced this command as docker cp exists for a while, so there no reason docker compose does also support this.

[evbo]

@ndeloof in docker cp you are copying data to/from a running container. Is the same true for docker-compose cp? Or does it persist the change to an image?

I'm hoping the latter is true, and that this can be configured inside the docker-compose.yaml file?

And what version of docker-compose is this supported under?

[ndeloof]

@evbo docker compose cp does the same as docker cp for service's container(s): it copy file into a running container. Nothing committed/persisted. This command was requested by many, and as an existing docker one I don't see any reason compose doesn't offer the same, with the same limitations (non persistent changes)

Also, this is command line only, there's no equivalent in compose file. Declaring a volume bind sounds a reasonable alternative if you want this set in your yaml configuration.

this is implemented in Docker Compose v2, not docker-compose

[evbo]

ok, thanks for clarifying @ndeloof

If I understand correctly, the main benefit is that docker-compose cp recognizes services (not just containers), so does that mean if I have a scale > 1 it would cp the data to each and every instance?

What if I have a scale: 0? No cp happens? Are there other benefits over docker cp?

[ndeloof]

this is indeed "just" docker cp applied to services for scale != 1 it will copy to all containers, or you can pass --index

[natotech]

What's the usecase? Most of the suggested usage I've seen were antipatterns.

One use case for me is to put the file custom.cnf into /etc/mysql/conf.d inside mysql container. Volume is not an answer to that but file copy would be more appropriate. So you expect me to create a Dockerfile just copy that one file?

[jadon1979]

Common Scenario: 1 or more files that live in non-empty directories need to be changed on-the-fly. You can't use volume with these directories as you'd lose all the other files in the directory. You most definitely do not want to copy the entire directory down to work around this. A suggestion of rebuilding the image every time you make a change completely ignores the real world, time, and the rage induced because you missed a comma. A basic copy command isn't an anti-pattern by any stretch.

copy:
  - ./some_dir/some_file:/etc/app/some_config/:<chmod>:<chgrp>:<chown>

A quick implementation of this might be an underlying volume that takes these files and creates a symlink to their respective destinations.

[ghost]

I just want to add to what @jadon1979 said, that any and all workarounds to the copy conundrum suffer in one of many ways:

• docker COPY command requires you own the image you're trying to copy into, else you need to create yet another image
• docker context (that you COPY from), requires everything to be co-located, typically forcing you to set the context to a parent directory in order to reach various disparate resources you may need.
• but as the context gets set to a higher and higher parent directory, the likelihood of unwanted files grows and so you must maintain a .dockerignore file, which can get pretty annoying especially when your cache is breaking because you forgot to ignore something new added to your context
• Then there's volumes, but their bind mount chowns everything to root, completely breaking a developer's access

There was recently a PR merged for docker-compose v2 that performs a docker cp. @shin- Just thinking outside the box here, but what if something like @jadon1979 compose syntax above supported this and each time you build the container it performs the cp for you? Could even extend this further to cp something out of the container after build too (very useful if you're using docker only for compiling a binary, etc.)!

[eldarj]

why not add this to docker-compose and enable for use in docker-compose.yml configs as well?

[XeJames]

I solved it. Gave up on Docker entirely and am using LXC. Solved everything for my use case AND no more people insulting me for "antipatterns". Win-win.

[alamar]

It is disingenous to write that the feature was implemented, because it was not.

The feature that was asked for have not been not implemented, and instead the feature that nobody has asked for and which was already fully covered by underlying docker was added.

The central issue of this feature request is that one does not own most of the images they have in their Docker Compose file, and when they bring up an image like grafana/grafana they cannot supply its config file in a hygienic fashion, and the container would not boot without that file making subsequent docker cp (or docker compose cp for all what it is worth) impossible. Bind mount of that file will lead to the config file hanging around in the host file system, which does seem like a huge anti-pattern WRT docker to me.

[washtubs]

https://github.com/compose-spec/compose-spec/issues/346

This ends the nearly 6 year long holy war. The reasonable among us were just asking for a way to put configs in our containers without volume mounts and without rolling new images.

Apparently it's available in docker compose 2.23.1. I would follow that thread if you are still seriously interested in this feature.

[ndeloof]

@washtubs while inlined config indeed don't rely on bind mount, this is not the reason this feature was introduced. We could support file-based configs without a bind mount, just this would have some side effects on existing users.

@alamar the example you mention suggest we should investigate lifecycle hooks. A pre-start hook to cp required file in container would solve this use-case.

[washtubs]

@ndeloof I haven't used docker compose in some time. I got renewed interest in it just now after realizing it was rewritten in go (yeah it's been a while), and seeing the output of docker compose up which honestly looks gorgeous, which is why I got curious to see if anything became of this issue.

So forgive me for my lack of ecosystem knowledge here. I assume those who are used to helm and kubernetes would like to have some kind of templating pre-processor engine on top of docker compose and use that to insert file contents inline. I'm not sure if compose has a "helm"-like sister software like kubernetes does, or perhaps even builtin support. But that's an approach that would be familiar to me which I would leverage to insert config files inline as a pre-processing step.

[ndeloof]

@washtubs docker compose do support variable interpolation to offer some flexibility, but we don't want this to become a templating system like the one helm uses. Obviously someone could built another tool to produce compose.yaml from another source.

[andoks]

@washtubs / @ndeloof : in a way docker compose already can be a lightweight templating tool for itself, by building your docker-compose config with variables, then "rendering" the complete result by using docker compose config :sweat_smile:

[washtubs]

That doesn't seem quite the same. I think that is basically coalescing multiple yaml files together into one, another thing that helm does. But helm uses go's template engine which gives a ton of flexibility on the kinds of things you can insert, and manipulations you can do, like quoting. A common pattern is inserting the contents of a file inline with indent. You can even define your own custom template functions to be reused throughout your project.

Honestly you could probably use helm to do this for compose, just make a helm chart alongside your docker-compose file, and you can use the helm template command to get the output docker-compose file with your file contents rendered and script the deployment yourself. (won't be able to use helm install or upgrade obviously, since those are k8 specific)

[sahlex]

Another usecase: I have a compose file on a local machine and my dockerhost ist a remote machine. How do I reference a file to be mounted from my local directory? Answer: not at all without building a specific image! A solution would be to copy it into the container.