allow removing something in docker-compose.override.yml

[stefanfoulis]

My usecase is that the base docker-compose.yml maps a certain port in the ports section. In my docker-compose.override.yml I'd like to change that port since I already have a different service running there on my host. As far as I understand the current implementation it is only possible to add stuff, override extends the other file. As far as I can tell there is no way to remove an entry. And I can't think of an obvious syntax for it.

[aanand]

Correct. Best practice is to not include anything in docker-compose.yml that you might not want to have in another environment.

[nkovacs]

My problem with your suggestion is that docker-compose.override.yml might not exist (it's an optional file the developer can create locally). I only want to use docker-compose.override.yml to locally customize ports, like this:

docker-compose.yml:

version: '2'
services:
    web:
        ports:
            - 80
            - 443
        ...

docker-compose.override.yml:

version: '2'
services:
    web:
        ports:
            - 32080:80
            - 32443:443

This isn't a big deal because this will only result in 80 being exposed twice, once as a random port and once as 32080. I think it would make sense in this case to only expose it as 32080 automatically (without having to specify anything in the override to remove the original config value)

[tomekit]

Facing the same problem. I've got one specific host which shouldn't start the one of the docker containers for given service. Yeah, maybe my docker-composer.yml approach is not the best, but I can see the need of removing something.

[paunin]

Did anyone get script to remove ports from docker-compose file ? :)

[paunin]

Alright, there is no solution so far, so I got a simple port-remover https://github.com/paunin/docker-compose-v2-ports-remover

In general I dont think docker-compose should allow to do it, as then logic of merging many files is gonna be complex...

[bluemoehre]

This was already solved here https://github.com/docker/compose/pull/3939

[jedie]

It's also not possible to removed a service, isn't it?

[OnkelTem]

Having an "override" thing that cannot do half of the job... well, it's something.

[mqu]

I'd like to be able to override some services and disable some others.

[sauron918]

Same issue, I'd like to have possibility to remove port, dns, etc in docker-compose.override.yml but not override or extend with another one..

[keithy]

Anyone notice that the only thing that you can't do with docker-compose.overide.yml, is to actually override anything!

[gnumoksha]

I would like to remove some services because I rarely use them on my machine, so I ended up with the following docker-compose.override.yml:

version: '3.5'

services:
  elasticsearch:
    image: hello-world

  selenium:
    image: hello-world

[darakian]

To add two more cents to the pile. It can happen that you pull a docker-compose.yml from another group and want to not run some of what they have turned on. Please add some syntax for removing.

[speller]

Three years passed and still nothing?

[immanuelfodor]

Stumbled upon this issue when trying to override the dns settings of a random Github project without modifying their docker-compose.yml. The new dns servers are appended to the original list instead of overriding the original ones - as I supposed it will work based on the file name. This behavior is misleading IMHO.

[Somebi]

Same problem here... It seems it's not overriding some values correctly. Can't remove either. For example i'm overriding with network_mode: hostand need to remove links, because otherwise it's throwing error:

ERROR: for processor Cannot create container for service processor: conflicting options: host type networking can't be used with links. This would result in undefined behavior

Setting links: [] doesn't help, links: null orlinks: ~ throws format error.

[m-hu]

If the above saying is true, it means that the inherent parser is not able to just load several dictionaries and then apply the overriding of keys following some arbitrary order, which is not complicated in nature.

The truth is that the current design has made a key dependent behaviour. ref https://docs.docker.com/compose/extends/

My feeling is that it would be better to go for a more intuitive behaviour for developers, like that in multiple inheritance cases. The current use case consists of combining multiple yaml files in the same command line, which looks just like a one layer inheritance. The better would even be able to support multiple layer multi-inheritance using some kind of inclusion key word. Also, to specify the case when child will either merge or simply replace, some key words like "super" could hence be interesting to have.

"Composition" could be a workaround to achieve the same thing as "inheritance". In the case of docker compose file one may do something similar. The key word "extend" becomes helpful in this case, which may , yet, result in a more verbose compose file. Also, one would need to very carefully craft each file so that things are always extended/override but not removed.

[thomaspeitz]

#!/usr/bin/env ruby
require 'yaml'

data = YAML.load_file "docker-compose.yml"

majorVersion = data['version'][0].to_i
serviceName = ARGV[0]
image = ARGV[1]

if majorVersion == 3
  data['services'][serviceName]['image'] = image
  data['services'][serviceName].delete('build')
else
  puts 'Version #{majorVersion} not supported yet. Please ask your cloud-ops team!'
  exit 1
end

File.open('docker-compose.yml', 'w') { |f| YAML.dump(data, f) }

Needed a small script which removes build and adds image to a docker-compose service. Maybe this helps other building CI scripts as well e.g. removing a service.

From my point of view the remove feature should be added to docker-compose overwrite as it decreases user experience of docker-compose a lot.

[keithy]

Hi Thomas,

I wrote a simple script, docker-compose-use, as the basis of my AngelBox toolkit. This makes docker-compose files "composable". You will find this at http://github.com/keithy/sprankle-pod and http://github.com/keithy/angelbox

[gothrek22]

@thomaspeitz or just use yq. you can easily edit parts of yaml with it.

F.e. Adding image: yq w -i ./docker-compose.yml services.nginx.image nginx:latest or removing ports: yq d -i ./docker-compose.yml services.db.ports

[sshishov]

It is ridiculous that the team of "professionals" of docker-compose cannot solve this easy issue for years. And even more funny that override file cannot even properly override anything.

[mlensment]

Any progress on this?

[typoworx-de]

I also vote for this. Already tried adding override_strategy: overwrite in the given yaml-section which won't work (anymore?!) mentioned here #2260

I would also vote for such an option which also would resolve deleting by setting a property to "[]" or "" to empty it.

[rekliner]

I'm still hoping this gets addressed. The best solution for me is using "image: hello-word" for unneeded services but it's so smelly. The only alternative is to ditch the override feature completely and duplicate much of the original docker-compose.yaml into a completely alternate version for production...not DRY or ideal for environment consistency but seems to be the standard practice.

[liamjones]

The best solution for me is using "image: hello-word" for unneeded services but it's so smelly

Option that might be slightly less smelly? - we tend to leave the image, etc as-is and do something like entrypoint: /bin/sh in the override file so the container runs sh then exits (rather than running its usual entrypoint/cmd).

[ptheofan]

It is ridiculous that the team of "professionals" of docker-compose cannot solve this easy issue for years. And even more funny that override file cannot even properly override anything.

Truly remarkable and only one of the many. Why not substitute it with a map instead? It's not like you can bind to the same host port twice. Why not a map where you could easily say, override it to "nothing" for example by doing 8080: -

ports:
     8080: 80
     21000: 5432

# ... in some other file extending the prev config
ports:
    8080: -

No wonder why people move so passionately to Kube.

[herbetom]

I just stumbeld accross this issue while searching a way to override a port. Kinda sad that this isn't possible. Semms to me quite essential.

[J7mbo]

Because this isn't yet implemented, I had to create to copies of docker-compose.yml, remember to use the correct one at any given time, and now update both whenever I want to make a change that is irrelevant to environment.

My use-case is elasticsearch and TLS. Dev uses a self-signed cert so needs a root CA and SSL verification mode set to certificate only, whilst Prod uses letsencrypt TLS set up during provisioning.

Here are my dev environment vars:

            - xpack.security.enabled=true
            - xpack.security.transport.ssl.enabled=true
            - xpack.security.transport.ssl.key=certs/key.pem
            - xpack.security.transport.ssl.certificate=certs/cert.pem
            - xpack.security.transport.ssl.certificate_authorities=certs/rootCA.pem
            - xpack.security.transport.ssl.verification_mode=certificate
            - xpack.security.http.ssl.enabled=true
            - xpack.security.http.ssl.key=certs/key.pem
            - xpack.security.http.ssl.certificate=certs/cert.pem
            - xpack.security.http.ssl.certificate_authorities=certs/rootCA.pem
            - xpack.security.http.ssl.verification_mode=certificate

Here are my prod vars:

            - xpack.security.enabled=true
            - xpack.security.transport.ssl.enabled=true
            - xpack.security.transport.ssl.key=certs/key.pem
            - xpack.security.transport.ssl.certificate=certs/cert.pem
            - xpack.security.http.ssl.enabled=true
            - xpack.security.http.ssl.key=certs/key.pem
            - xpack.security.http.ssl.certificate=certs/cert.pem

If I try and run prod with any of the ssl.certificate_authorities vars, even as empty strings, elasticsearch will not start. They should not be empty strings, they simply shouldn't exist.

Given this use-case, I now have to have a dev docker-compose and a prod docker-compose, which are completely identical bar the above change in environment variables.

It would be very useful to not have to do something like this.

[mschwartz]

My use case for this missing feature is to have development and production setups.

By default, I want to just type

docker-compose up

and be in development mode. This bind/mounts the source code over the application within the container so I can edit and see the changes happen within the container immediately.

When I go into production mode, I (think I) have to do something like this:

docker-compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.production.yml up

And the production.yml would remove the binds.

There are probably better ways to achieve this, but this is a real world use case.

I mean, I could have a .development.yml file that I always have to use in development and if I don't it's production. But then I'm typing the -f x3 all the time, when the short docker-compose up is what I'd like to type 99% of the time.

[keithy]

You need to have a look at my docker-compose-use script see github.com/keithy/sprankle-pod a more advanced form is available in https://github.com/keithy/angelbox

[mschwartz]

Yeah, sh scripts or aliases can soften the sharp edges, but it's still not the same ;)

[keithy]

its pretty close you can type: docker-compose-use whereas a built in would be docker-compose use You can use the COMPOSE_FILE environment variable in the .env file to achieve what you wish. The use script just makes it possible for that .env file. to be composed of several pieces.

[nick-brady]

My use case is I wanted to mount /etc/letsencrypt in docker-compose.yml and have my docker-compose.override.yml remove the bind so that when a developer types docker-compose up, they are not trying to mount /etc/letsencrypt. Essentially the same as https://github.com/docker/compose/issues/3729#issuecomment-606236585. I feel mine would be a pretty common design pattern so it was worth mentioning.

I will end up creating a docker-compose.production.yml which does nothing but add the bind mount because of this. Not the biggest deal, but definitely leaves a sour taste. Would appreciate definitive feedback from the docker team on plans here, though silence I suppose is answer enough.

[mschwartz]

SUGGESTION:

   web:
      remove:
          volumes:
              - /the/one:i/want/to/remove
      volumes:
         - /the/one:I/want/to/add

Anything in remove: is removed and then anything else is added/replaced/whatever.

[nick-brady]

Looks like should have done more research. Thank you for documenting that in this thread!

[mschwartz]

No, that's a suggestion!

It's not implemented that I know of.

[masto]

This is clearly shouting into the void as evidenced by four years and several people actually writing the code and getting no response on their PRs. But I just ran into this myself so I'm adding my voice to the chorus. Why not this, why not that? My use case is to take a pristine docker-compose.yml that comes from an upstream package I don't control, and put only a minimal set of changes in my production override file to adjust it for my environment. One of those is to delete port mappings I don't want.

Of course, I am capable of solving this in other ways (and have done so to unblock my project) but it introduces more maintenance hassle and additional complexity on top of a system which already has a mechanism for layered configuration.

[hubertnnn]

@mschwartz The remove key looks nice, but its still quite limiting (it allows you to remove a component from the service but not a service as a whole). How about something like this (I assume ! is not allowed in key names since those should be alphanumeric, so there should be no conflict):

docker-compose.yml:

version: '3'
services:
  web:
    image: nginx
    ports:
      - 80:80
      - 443:443
    environment:
      - FOO=foo1
      - BAR=bar1
      - BAZ=baz1
    volumes:
      - .:/var/www/html
  useless:
    image: something-useless

docker-compose.override.yml:

version: '3'
services:
  !useless: ~       # remove entire useless service
  web:
    !volumes: ~    #remove all volumes
    !ports:            # replace entire list of ports with this
      - 8080:80
    environment:
      - !FOO=abc # replace one env variable
      - !BAR         # remove the other one

This will result in following merged yaml

version: '3'
services:
  web:
    image: nginx
    ports:
      - 8080:80
    environment:
      - FOO=abc
      - BAZ=baz1

The idea is that ! would mean "remove everything in this key/position and if value is provided then insert this value instead"

[hinorashi]

any update on progress guys, btw pls stay safe during covid pandemic :whale:

[rsjohn]

SAD!

[signalwerk]

@rsjohn Welcome 👋 This is an open source project and we all do our best to keep the spirit high! it's always awesome to encourage and to contribute in an open community. I noticed your comment and would like to invite you to actively participate in the discussion and/or codebase. All opinions and inputs are always welcome. But please consider; people here are often working on a voluntary base to provide you with an awesome product that works out of the box.

Do you have time to take on that Issue to provide a solution? We would love to have a Pull reques on this to get it off the table? That would be super awesome!

[djbios]

Any progress?

[archfz]

I mean this really should be an easy feature to implement :/

[gnat]

Encountered this as well- it's an obvious feature that many people need.

Added more .yml files as a lame workaround; the hello-world hack for entire services is hilarious and shows how stupid the lack of this feature is.

[vschoener]

Came across here to remove volumes from override in a specific case I don't need them and unfortunately, we can't.

[gnat]

Came across here to remove volumes from override in a specific case I don't need them and unfortunately, we can't.

This was my exact use case too actually, to remove unused volume mounts.

[Nielio]

@thomaspeitz or just use yq. you can easily edit parts of yaml with it.

F.e. Adding image: yq w -i ./docker-compose.yml services.nginx.image nginx:latest or removing ports: yq d -i ./docker-compose.yml services.db.ports

In case someone wants to use this in a dind gitlab-runner. You have to install it and use a slightly other syntax. Here is a working example for removing a service from a compose from a git submodule.

variables:
  GIT_SUBMODULE_STRATEGY: recursive
  STACK_DEV: blabla
  SERVICE_COMPOSE: blabla/docker-compose.yml

stages:
  - compose

compose-develop:
  stage: compose
  before_script:
    - apk update 
    - apk add git jq py3-pip
    - pip3 install yq
    # pull sub repository
    - git submodule update --remote
    # remove traefik from service compose.yml
    - yq -y -i 'del(.services.traefik)' $SERVICE_COMPOSE
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
  script:
    - docker stack deploy -c $SERVICE_COMPOSE -c compose-dev.yml --with-registry-auth $STACK_DEV

[dm17]

The two main use cases I see in this thread, which I also have, are prod vs dev (certs -vs- no certs/self certs). In my environment, basically all of this stuff is contained in docker-compose.yml's '-labels'... So, perhaps there a way to run a docker-compose disabling the input of all labels?

Adding support, thanks! :)

[PhilippeFerreiraDeSousa]

Hi, that is a lot of salty comments. Is there a branch / PR open for this feature or nothing at all?

[umstek]

I'm new to docker.

Ended up creating a base docker-compose.yml with everything common, docker-compose.dev.yml for dev, docker-compose.prod.yml for prod, so no docker-compose.override.yml involved. And created a shell script to start these e.g.:

#!/bin/bash

case $1 in
prod)
  docker-compose -f docker-compose.yml -f docker-compose.prod.yml up --build
  ;;
*)
  docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --build
  ;;
esac

What are the caveats of this approach? (Haven't encountered any yet.)