Add `copy` to the yaml configuration

[riquito]

I want to deploy different services using the same image but with a different configuration file. Currently to achieve that I can

• build as many images that inherit from a common image plus a different COPY in each
• mount a single file volume

The first solution isn't feasible and the second it's overkill. I just want to copy a file, not keep it in sync

Proposed configuration:

myservice:
    image: foo/bar
    copy:
        - src:dest

[prologic]

What is the semantics of copy here? Are we copying files from the host where docker-compose is run from into the newly running container using docker cp?

[prologic]

AFAIK there is no current way of doing this:

See:

$ docker cp -h

Usage: docker cp [OPTIONS] CONTAINER:PATH HOSTDIR|-

Copy files/folders from a PATH on the container to a HOSTDIR on the host
running the command. Use '-' to write the data as a tar file to STDOUT.

  --help=false       Print usage

[riquito]

What is the semantics of copy here? Are we copying files from the host where docker-compose is run from into the newly running container using docker cp?

Yes

AFAIK there is no current way of doing this:

I see that the docker cp command is unable to do it. Yet it's still possible by accessing the container data on the host side (which I ammit is ugly)

http://stackoverflow.com/a/24805696/210090

[prologic]

Yeah I'm not going to advocate we do this at all. This will break a lot of things and assumptions about Docker. This is also highly dependent on the underlying storage driver being used as well. Docker is meant to be agnostic. I think we should a different solution to your problem :)

[ljrittle]

Now that docker-1.8 added support for copy into a container via docker cp and swarm-0.4.0-rc2 added support for docker cp, it would be great to revisit this issue at the compose level. Here is the use case (which mirrors what we actually do in almost production now):

A docker-compose.yml file that mentions many containers by CI-built image tag (i.e. we don't use build: in production; maybe we could now but it didn't play nice with swarm in past releases) that all need e.g. static hadoop configuration files mapped in which exactly match the deployment environment. Currently, one must manually sync the directory in which docker-compose.yml resides to the exact path on each target docker-machine in the swarm. Then one adds a stack of volume: lines.

Supporting docker cp would allow the removal of a custom configuration file sync in our system deployment method and allow better control of when configuration changes are injected (since anyone changing the files mentioned in the volume: lines are affecting all prior deployed containers (and whatever time the internal implementations re-read said configuration, perhaps only upon the next container restart, perhaps when various tools inside the container start; which may be unexpected) and future (which is expected).

OTOH, (as mentioned briefly above) maybe we should use build:. The downside here is that we need to write an extra Dockerfile per deployment container type. One for the CI-built images and a second for the static configuration file injector. copy: in compose backed by the recent changes to docker cp would neatly allow us to avoid all this duplication.

[dnephin]

It still seems like using a volume is a better way to do this. It doesn't have to be a host volume, it should work with a data volume container.

[prologic]

Bedies docker-machine has docker-machine scp anyway which you can do with the preparation of the machine; and then you can do host volumes the normal way eith docker or docker-compose

[ljrittle]

"It still seems like using a volume is a better way to do this. It doesn't have to be a host volume, it should work with a data volume container." I fail to see how that is best when the target is a swarm. AFAICS, one has no way to express that a data volume container should exist at every possible swarm node.

[prologic]

Can't you deploy a data volume container on very node via swarm itself anyway like you can with any other container ala scheduling strategy?

[ljrittle]

"Bedies docker-machine has docker-machine scp anyway which you can do with the preparation of the machine; and then you can do host volumes the normal way eith docker or docker-compose" I currently do this. It is painful. People managing deployments forget to to move the configuration files. If it was supported in docker-compose yml then it would happen whenever a container was recreated without the manual step(s).

[ljrittle]

"Can't you deploy a data volume container on very node via swarm itself anyway like you can with any other container ala scheduling strategy?" Maybe we could but I admit that I don't see how in light of scaling.

[ljrittle]

"Can't you deploy a data volume container on very node via swarm itself anyway like you can with any other container ala scheduling strategy?" Humm, if I know that my swarm is size 10 nodes; can I just scale the data volume container to be size 10 with a policy that no dups are allowed on a given docker engine node? The problem (unless I have missed something) is that e.g. containers that need to reference that data volume have no way to --volume-from a container with an index that matches. Guys, I have been looking at this problem for 2 months now. The best I could do is create a script that uses docker-machine scp. But it is a manual step upon after editing a configuration file and before docker-compose up. It also requires that one can write the exact path on the target machines of the swarm as the root of the docker-compose.yml (i.e. it smells like a major hack).

[prologic]

Perhaps my efforts in factory can help automate this process of spinning up docker machines and swarm clsuters with the "right" data sitting there ready to use? (_albeit it's still in early development and I'm using it to combine docker-machine and docker-compose in a single setp_0.

[dnephin]

copy is an operation, not part of the service configuration, so it doesn't belong in the compose file.

Docker 1.9 is adding a new volumes api, and volume drivers are in already. Volumes are really the correct way to support this. Not trying to copy files around.

Thanks for the suggestion! but I don't think this really fits with the design of compose right now.

[dimitrypo]

copy is an operation, not part of the service configuration, so it doesn't belong in the compose file.

What if one want to copy just a configuration file? (to a docker swarm, so volume_from wouldn't work)

I have exactly the same situation, want to copy custom mysql, nginx, haproxy etc. config files, we have clusters in production, and instead of just using one copy command to copy configs from local to remote

I have to use images, build, pull each time, while there are no need whatsoever and just copy command will save lot of deployment and development time

[johnculviner]

This is very surprising to me that more people don't have a problem with this not being readily scriptable. Am I thinking about docker wrong? I think my solution now would be to use ansible to copy out the config files to the host and then volume mount from the host to the container(s) that need it

[unleashed]

Hitting the same issue. I am reusing a Dockerfile for N different services and I don't want to create N different Dockerfiles with their own COPY command or create a volume just for that.

[dimitrypo]

Btw I'm currently solving that by running dozens of lines of sed commands - http://unix.stackexchange.com/questions/112023/how-can-i-replace-a-string-in-a-files

Maybe not optimal solution but works considering that docker-compose doesn't support copying files.

Btw mounting and copying of course possible but it goes against automatization docker-compose offers - if I would prefer to do everything manually I wouldn't use docker-compose, wouldn't I?

[johnculviner]

The more I've dug into Docker the more it seems like a lot of this tooling is good for "hello world"/"look how fast I can spin up this cool thing with\ shell\ commands\ like\ this\" on a single machine type use cases (docker-compose included). If it gets much beyond that things start to become quite complicated and IMO fall apart a bit which is where traditional provisioning tools can come in and save the day when used appropriately.

I've found using Ansible to set up machines in an idempotent way to be very easy when I have machine specific volumes, files and templated config files that must exist for the docker image to start. Then at the end I can use Ansible to invoke docker-compse or even use the Ansible docker module whose syntax is nearly the exact same as docker compose with some nice added bonus features.

At the end of the day the above approach allows everything to be scripted, idempotent and most importantly checked into source control.

[ColinHebert]

@dnephin some people above suggested the use case of CP being part of the configuration, would it make sense to revisit the assumption that CP is only for operations and not for configuration.

A simple example is creating a container, copying the configuration across in the correct place, starting the container once the configuration has been copied.

This allows the creation of images that do not embed the actual configuration and only deploy the configuration as the container gets started.

[dmikov]

Definitely will be very helpful. I am working with postgres image as database back end now and I don't want to rebuild it, just to update a script in /docker-entrypoint-initdb.d folder.

[ctindel]

Why don't you guys re-open this issue until the feature is implemented or a workable solution is found that people like, since nobody here likes the suggested options?

[NicklasWallgren]

This issue should be re-opened, as @ctindel mentioned.

[metmajer]

+1 for having the issue reopened

[lhridley]

+1 for opening as well.

[ghalusa]

+1 for reopening.

[dixiedream]

+1 for reopening.

[pbrueske]

+1 for reopening.

[AnthonyWC]

+1 for reopening.

[GeekHare]

+1 for reopening.

[BorzdeG]

+1 for reopening.

[michaelarnauts]

What's wrong with a read-only volume?

myservice:
    image: foo/bar
    volume:
        - src:dest:ro

[koooge]

@michaelarnauts see this.

[tonivj5]

👍 for reopening

[nguyenphuocpham]

+1 for reopening.

[andresgaragiola]

+1 for reopening.

[Orfen-0]

+1 for reopening

[ericlery]

+1 for reopening

[jayaskren]

+1 for reopening

[matthewh]

+1 for reopening

[legovaer]

+1 for reopening

[LiberQuack]

+1 for reopening

[jasonterando]

Here's my rationale for allowing a copy in Compose.

I have an application that has runs on numerous Containers. For sake of argument, let's stipulate they all run Apache. They all have DocumentRoot definitions pointing to things like "/var/www/partX". The Dockerfile knows nothing about the contents of the Apache configuration, and the Compose defines the volume reference for /var/www/partX. This works great for development, because I can debug and tweak code without modifying container contents.

The rub is when I want to deploy. One of the main attractions about Docker is that I can deploy a container that is its own universe. In order to get everything in the container, I have to copy the files, which means that I would have to have separate Dockerfile and Compose definitions from what I use for development. On a large project, this means I have to maintain two sets of definition files, which introduces more possibility for errors.

My preference would be to maintain a single set of Dockerfiles, and do the orchestration on the Compose file. Compose would be where I define if I am going to set up /var/www/partX as a shared volume or if I am copying files into the container.

Deploying with shared volumes seems like a can of worms. If I am updating an existing version in production that depends on a shared volume, I can't update that shared volume without setting up a maintenance window or coming up with a scheme for versioning my shared volume. I am tightly coupling containers with external files, and if I'm introducing that type of complexity, I am negating some of the benefit of using Docker.

It seems like I could script changes to the Dockerfile definitions to conditionally copy files, but it seems more "logical" do handle all of my Docker logic in the Docker universe, and Compose seems like a logical place to do it.

[ugurcanlacin]

+1 for reopening.

[kalinchuk]

+1. Would be very helpful to copy a configuration file in docker-compose.yml file instead of Dockerfile.

[DimYurich]

+1. Just ran into it trying to supply a config file for the service. Read-only volume seems like an option, but would rather copy it.

[raszi]

Simply mounting a volume could do the trick:

version: '2'
services:
  foobar:
    image: 'postgres:9.6-alpine'
    volumes:
      - './docker/schemas/foobar:/docker-entrypoint-initdb.d'

[jasonterando]

@raszi - Yah, the mounted configuration is the direction we're going to go for now. One of the things that I really liked about Docker though was the notion of a self-contained container that I could run integration tests on, without any concern for external dependencies (like linked configuration). If I am doing blue/green or A/B deployments, and there are differences in the configurations used by different versions of containers, things can get a little fragile. I can get around this by doing things like versioning configuration files in a shared volume of each of my environments, but it just seems more complex or brittle than it could be.

"Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment." To me, that ideally would include configuration. Compose is sort of doing a "build/compile and run", I think some of us here are wishing Compose could do a "build/compile, link and run". It is not how it works today, and I can understand the lack of enthusiasm for going that route, but hey, it's a forum, it doesn't hurt to ask ;)

[raszi]

I agree, it would be great to have this feature. I just posted my solution to help others.

[MeGaPk]

+1 for reopening.