Define services which are not started by default

[bfirsh]

Users quite often define maintenance scripts, test suites, debugging systems in their Compose files which they don't want to run when they do docker-compose up.

There should be some way of defining what services are started by default, but can be still be run manually by doing docker-compose up servicename or docker-compose run servicename ....

Possible solutions

1) Recommend users to use a separate Compose file 2) Add an option to services to make them not start by default 3) Add a top-level configuration option to define the default services 4) Add a concept of a thing like a service, but is just for one-off commands ("scripts", "tasks", etc...)

(Please suggest others if you have ideas.)

Data points:

• https://github.com/docker/compose/issues/697
• https://github.com/docker/compose/issues/912
• https://github.com/docker/compose/issues/942
• https://github.com/docker/compose/issues/1439
• https://github.com/docker/compose/issues/1547

• 542

• "test" service in https://github.com/heroku/logplex/blob/master/docker-compose.yml
• @cpuguy83: "I've got a little helper service in my compose yaml that injects a bunch of stuff into redis for hipache, which of course exits when it is done. Can't use compose up w/o the -d" this is fixed now that we don't exit from compose up until all services have exited.

[dnephin]

+1 for option 1

I think adding services that aren't actually part of a composition, but just happen to need to link/attach to it, is a bad design choice. There are a few other use cases that would be solved by allowing some form of includes syntax. Solving multiple problems with a single feature is always nice.

Some of these issues (the ones that deal with data-only containers, #942, the last comment from @cpuguy83) are actually already fixed by #1754, I don't think we need to consider them a problem any more (after 1.5).

[kojiromike]

We develop Magento extensions. To do that, we need a simple way to run a webstore on a LAMP stack. Compose makes that easy. But we also want to run phpunit, various static analysis tools, documentation builders, etc. In fact, the vast majority of our "services" are just commands, like docker-compose run --rm phplint. Ideally, an unqualified command like

docker-compose up -d

would only start long-running service (actual services as it were), and would not trigger other effects like phpunit, unnecessarily. (It's also important that it not trigger things out-of-order, like running Selenium tests before the web service is up.)

All we are really trying to do is take the pain of environment management away from developers, not run production services. I don't know if @dnephin 's comment is representative of the direction Compose is headed or not, but it seems ambiguous to me whether Compose is still planning to serve the niche Fig started out in. (Because, to be clear, I think option 1 does not support the ease-of-use that developers need to use this as Fig apparently intended.) I totally understand if Compose doesn't want to have multiple responsibilities, but I hope that someone can let us know, clearly, if those of us using this product for fast, isolated development environments should head in another direction.

[ryneeverett]

@kojiromike, what is it about (1) that is unsuitable for your use case? Is it just about the command semantics (docker-compose run --rm phplint vs. docker-compose --file phplint.yml up)?

[ppg]

Its that docker-compose up -d tries to 'start' phplint and docker-compose ps reports that that the phplint 'service' is down. In reality that's not a service container its a tool container, it should have no concept of up/down. It was my understanding that tool containers are embraced by docker (its how you would run something like redis-cli, clearly not a 'service') and while I use them more in development, I think they have a place for production too; like why have say awk installed on production machines or in containers, why not run it through a container with linking to get predictable behavior out of it.

[chrisguidry]

:+1: I routinely find myself wanting to create a tests container alongside my other services to encapsulate running unit tests. My "workaround" is to set the command to "/bin/true" and run the container with the unit test command specified on the CLI. Being able to specify which containers should start at up time would be great!

(btw, nice work all around, folks)

cc @jameydeorio

[kojiromike]

@ryneeverett The semantics are one part of it. It's a problem of self-documenting and findability. Currently, we tell developers docker-compose run --rm foo bar. We invite them to create a shell function or alias, but we don't maintain a standard alias/rcfile for projects. (We don't want to standardize things outside of containers; we want to use Docker to standardize.) Adding a new file for some commands creates a hierarchy of importance: The docker-compose.yml file becomes the "default" file for important things, and the "other" file becomes less important.

The other thing is just maintaining relationships between services becomes more onerous. Just because we want something not to run by default doesn't mean it doesn't use a link or volume from a service that is long-running. extends doesn't actually provide all the facilities we would need to link services to "commands" (one-run services). Even if it did, if we have to use multiple yaml files, we will be forced to use extends where we otherwise wouldn't need to.

[ryneeverett]

extends doesn't actually provide all the facilities we would need to link services to "commands" (one-run services). Even if it did, if we have to use multiple yaml files, we will be forced to use extends where we otherwise wouldn't need to.

@kojiromike That's what I suspected. I wonder if you would be satisfied with improved extends support + some way to do subcommands (which would be functionally identical to extends) within a single docker-compose.yml. But perhaps the latter is the same as (4).

[tomfotherby]

2) Add an option to services to make them not start by default

I vote for Option 2. e.g. something like a start: false directive. The advantages are that we avoid the need for multiple compose.yml files or extra config files, you can just read one compose.yml to get the feel of the whole app stack.

[dnephin]

I think our proposed solution for https://github.com/docker/compose/issues/1987#issuecomment-139632238 will handle this. "admin" services can be defined into a separate config, and added with -f when an admin command needs to be run against a composition.

[sherter]

@dnephin The solution in #1987(comment) does handle "admin services", but it doesn't handle "data only containers", right? You would still have to use the command: /bin/true workaround.

[cpuguy83]

You shouldn't need data-only containers with compose as compose will handle swapping the volumes to re-created containers for you.

[sherter]

@cpuguy83 In the following example the data container is not really necessary, but if I want to change the volume, I only need to look at one designated place.

nginx:
  image: nginx:1.9
  volumes_from:
  - data

php:
  image: php:5.6-fpm
  volumes_from:
  - data

data:
  image: debian:jessie
  volumes:
  - ./:/app

However, I agree that this is a corner case and adding additional syntax is probably not worth it. I'm just saying that the proposed solution doesn't handle this case.

[dnephin]

True, some command is still required for data volumes, so you would have to use a minimal image to support that command.

I'm not completely up-to-date with the new volumes API that's coming out, but my hope is that we will be able to add a volumes: section to compose which will handle the data volume in a better way (instead of requiring a container for them).

[CWSpear]

This doesn't necessarily make it the right choice, but I think learning curve/ease of understanding should be considered.

I feel like (2) is the easiest to comprehend. I don't really have a way to confirm this, but my gut says most people who aren't intimately familiar with all of docker-compose's options that run into the problem we're trying to solve here say, "I wish there was some way to get that container to not start when I run docker-compose up," and they see start: false and bam, we're done and happy.

They don't say, "If only there was a way I could create a 2nd file with an awkward linking story to solve this..." (tho I realize that https://github.com/docker/compose/issues/1987#issuecomment-139632238 helps with the "awkward linking story," ya?).

(4) was kind of vague, but a dedicated avenue for the way for scripts and one-offs like this fits this "makes sense" bill.

[kalbasit]

Today I was looking for exactly (2) but ended up with my least favorite solution, a second YML file. My use case:

I have couple of container and all of them link to the same mongo container. I'd like to offer myself and the team the ability to load fixtures into the mongo database and I figured the easiest way to do that is to load the mongo container under a different name fixtures that itself link to mongo and then run mongorestore --host mongo --port 27017 --drop /dump. Since we don't want to load the fixtures at all time, it felt natural to have it with a start: false but ended up having a separate YML file for both containers fixtures and mongo.

Works well but start: false is much cleaner IMO. If I would have 10 or more permutations of this so-called fixtures container then yes start: false would be a bad idea and I'd go with (1).

[tonivdv]

Just had to make a compose file where some services should not be executed with the docker-compose up -d command. For me option (2) would be the best solution to my problem.

[mgriego]

I've run across this as well. Not sure which I think is the best solution, but it what it comes down to for me is that I need the ability to build images that are utilized as part of a composition. They're for ephemeral containers, but I need the image to be ready for use so I can run the container on demand.

[qcho]

Hi! same issue here, glad there is a nice grouping issue, great work!

In my case I have a python stack working with a couple of containers with services. Everything is working fine, but I've just found a case out of the stack. I'm managing static dependencies with bower, and want to run the bower command inside a container, it's just a script that we run once a while.

Would be great to run this script within the compose structure we have. I imagen something like: docker-compose run node bower install

So I like the 2th of 4th option. Just need the service not to start, it's not needed :P

If there is some consensus I could try to send a pull-req for something like "restart" start: always maybe... dunno.

[jmreicha]

Another vote for option 2.

[gittycat]

I would prefer option 2 as well. The new Docker Data Containers can be specified in a docker-compose file but they don't need to be running to be used.

[snussbaumer]

Option 2 for me too

[philcal]

Option 2 would be great.

[IAmJulianAcosta]

+1 for option 2

[xaka]

+1 for option 2. Maintaining multiple files and typing their names is just wasting of time. One single boolean flag would rule them all. :beers:

[pmoust]

+1 for opt 4, If I must pick a second favourite its opt 2.

[alexsapran]

True that the best options is 2,4 if i can +1 both then thats what i do if not I am going with the majority and vote for 2.

[aris-b]

Voting for 2

[stafot]

both 2 and 4 are nice to have. +1 for 2

[sherter]

I don't think it's necessary to introduce a new concept for tasks. These read very well in my opinion and are possible without many modifications:

// 'task' being a service that manages all my tasks, probably with some custom entrypoint script
docker-compose run task tests

// 'tests' being a service specifically for testing with the default command set to run my tests
docker-compose run tests

The current "solution" with the separate compose file is not really a solution in my opinion. As @xaka said, nobody wants to type all this:

docker-compose -f default-file.yml -f additional-tasks-file.yml  run task myTask

What you'll end up with is a ./run-task script that adds all the boilerplate before myTask for you. But now you have added another entrypoint and interface to your multi-container-app. Developers see docker-compose.yml and think: "Oh great, a compose app. I know how to handle this thing!" And now you have to tell them: "Right, you can manage it with docker-compose like every other compose app... oh, but wait... there is also this additional script you need to know..."

start: false / up: false / some-additional-flag-to-a-service: false is probably the simplest thing to implement, but most likely also the clearest and easiest to understand. And it would improve usability so much.

[mgriego]

what @sherter said. :arrow_up:

[ThiefMaster]

:+1: for that. separate dockerfile is a huge pain especially when networks are involved

[gittycat]

The start: true|false approach is too limited. What if some services are used for testing, some others for admin and the rest for normal operation?

I would prefer adding the notion of grouping to services.

    myservice:
       group: `admin`

If the group attribute is not defined, default is assumed. That way we could start the default and admin services using docker-compose up -g admin -d.

Better yet, make groups an array.

[kojiromike]

Creating group-classes of services seems like it would be a powerful feature, but it also seems tangential to this issue.

[qcho]

On docker-compose version 2, every container is declared inside the services: top-level-item. Declaring a service, with start: never to run a script sounds wrong. So considering the new format, shouldn't we declare an extra top-level item apart from services, volumes and networks?

My proposal:

• add a new scripts:, transients: (or think a better name) top level item to v2.
• inside transients you can define a container as you would do with any service
• the only difference is that they are not going to start by default, since they are intended for transient use only.

Example:

version: "2"

services:
  web:
    image: myapp
    networks:
      - front
      - back
    volumes:
      - /usr/src/app/
  redis:
    image: redis
    volumes:
      - redis-data:/var/lib/redis
    networks:
      - back

scripts:
  bower:
    image: my_image_with_bower
    volumes_from: web
    working_dir: /usr/src/app/static
    command: "bower"
// maybe would be great to place something like "bower $@"
// to define where you want the cli arguments to be placed, by default at the end.

volumes:
  redis-data:
    driver: flocker

networks:
  front:
    driver: overlay
  back:
    driver: overlay

And then you could run:

docker-compose script bower <EXTRA_CMD_ARGUMENTS |  default nothing>

docker-compose script bower install

Concerns:

• It sounds OverPowered to create another top level item just for this purpose of managing start, but it's derived by the fact that the new format is declaring the behaviour in the top level.
• I'm trying to imagine extra flexibility (and problems) this may add.

Lastly, on the groups feature, sounds nice but if you have that many grouping I don't see the problem on creating docker-compose files for each one of them. Probably the feature that you want is docker-compose file inheritance, that would be awesome!

PD: @bfirsh maybe if you like the idea you can add it to the "Possible Suggestions"

On second though this is a revamp of the 4th suggestion because of the imminent new format declaring services.

[kwiky]

Voting for option 2 : Clearest, easiest to read, and don't need to learn new concepts Just want a service that don't start by default

[cpuguy83]

What about something like disabled: true

[kojiromike]

@cpuguy83 That would seem to imply the whole service is disabled, even for run. I'd find it confusing.

@qcho hmm, now that I've refamiliarized myself with Docker since 1.6, I can see what you and @gittycat are talking about. In that sense, I really like @gittycat 's approach. I picture (blue sky) an interface like:

groups: # if no groups, all services are in the "default" group by…default
    - foo
    - bar
services:
  foo:
    image: imagine
    groups: foo
  bar:
    image: energy
    groups: bar
  baz:
    image: money
    # no groups, so "default"
   quux:
    image: word
    groups:
      - bar
      - default

Meanwhile, in the shell…

docker-compose up -d # Up all services in default group, so 'baz' and quux
docker-compose up -d --groups foo # Up all services in the foo group
docker-compose up -d --groups foo,default # You get the idea
docker-compose run --rm bar somecommand # We never started 'bar', so it can be a one-off command

An approach like this would be awesome and obviate the need for this ticket, but does go beyond its scope.

[cpuguy83]

@kojiromike I don't think so. This is how init systems refer to services that shouldn't start up automatically.

[cpuguy83]

It's also simple and any "confusion" can be resolved with documentation. I also find it much less confusing than this grouping, which is completely unrelated to the concept of a service starting.

[kojiromike]

@cpuguy83 I think the semantics of "services" is exactly what is confusing in the first place. I agree that the grouping thing is scope creep. In that sense I prefer option 4/@qcho 's approach, where they clearly differentiate "services" from "things that are not services".

[qcho]

Thats the point, why should I put a container that will never run a service under a "service...disabled" category. Sounds like an ugly patch someone made and It's not intuitive.

Maybe the "version 2" of the format should have taken this issue into account. For example another spec could be

version: 3?
containers:
  webserver:
    ...
  database:
    ...
  cache:
    ...
  some_script_container:

services: (they are groups):
  production:
    webserver:
      ...
    database:
      ...
    cache:
      ...
  development:
    webserver:
      ... DEFINE SOME CUSTOM DEV STUFF ABOVE basic container definition
    database:
      ...
    cache:
      ...     

Ok now we have proper services, with groups definition. I can start production or development service group. Or just run a script in the some_script_container. since it's not defined in any services no one will be started

[cpuguy83]

@qcho Depends on the definition of a service, but in any case -- I'd call into question something which should only be run on occasion, and the only a human can determine if it should be run or not.

So let's say compose adopts something like a job object.

1. jobs only get run once
2. jobs are started after services
3. jobs are expected to exit
4. ???

Now compose also needs to keep some local state about the job, which it currently doesn't do at all.

[mgriego]

Which is why adding a simple auto-up: false or autorun: false is the easiest and least creepy (scopewise) way to handle this.

[qcho]

@cpuguy83 I think you are extending what I'm saying with something more complex. I don't intend docker-compose knowing about any job or workflow. External scripts can be made for that. What I intend is that docker-compose should define the entire stack for an application to run.

I can Imagine myself creating a script and keeping the state you are saying. Even calling the script before service start, not just after. What I don't imagine is myself parsing the composer file because I need to check what volumes I need to import, or what configuration i need to extract from it on an unknown container to run some script that applies to it. So I think docker-compose should give me that container; running the script and keeping state is my problem.

I don't see why people keep saying that

services:
  my-script-container:
    auto-up:false

Is simplier than:

scripts/containers/transients/you_name_it:
  my-script-container:

It's the same level of complexity. But less hacky semantically.

[jimzucker]

To get the ideas on one thread, reference #2803

Use case: You have a project with many components and users what to pick and chose which to install but he compose file installs them all.

Proposal: We add a option to put int he docker-cmpose.overrider.yml to exclude a image defined in the docker.compose

ie

some-image: exclude: yes

The behavior would be to ignore that entry in the docker-compose.yml

Can you let me know what the team thinks, I would be in interested in making the change.

[KiraTsunayoshi]

Voting for #2 as well...just ran into this need today.

@jimzucker's proposal would also work, though I like the thought of seeing "start: false" in the main .yaml file to immediately clue you in that this 'service' won't run unless you call it explicitly. Otherwise you (or in my case the end-user/developer you handed the docker-compose files to) need to remember to look for an override file.

[LLFourn]

+1 for 2. I have a situation where I need to build a docker image as part of compose but it doesn't run as a service. The other services (which have docker sock mounted inside them) run containers from the image every so often.

[niko]

+1 for 2. And +1 for "auto-up" or "auto-run" as wording.

And can't the cases for groups of services (as explained by @gittycat) be handled via environment variables ala "auto-up: ${ADMIN}"?

[acran]

I also see real use cases to mark services in the docker-compose.yml to be not automatically started with a simple docker-compose up but instead to be started only explicitly.

Solution 1) is one possible way now but in my opinion too cumbersome since one has to specify one or more yml files instead of just calling docker-compose up and having to split or even duplicate the files.

Since I'd really like to see something like solution 2) (as many others do too) I implemented a proof-of-concept of this in #3047 so you can play around a bit with it to see whether it could be a viable solution.

[xcash]

+1 for option 2