allow removing something in docker-compose.override.yml

[stefanfoulis]

My usecase is that the base docker-compose.yml maps a certain port in the ports section. In my docker-compose.override.yml I'd like to change that port since I already have a different service running there on my host. As far as I understand the current implementation it is only possible to add stuff, override extends the other file. As far as I can tell there is no way to remove an entry. And I can't think of an obvious syntax for it.

[dm17]

I'm new to docker.

Ended up creating a base docker-compose.yml with everything common, docker-compose.dev.yml for dev, docker-compose.prod.yml for prod, so no docker-compose.override.yml involved. And created a shell script to start these e.g.:

#!/bin/bash

case $1 in
prod)
  docker-compose -f docker-compose.yml -f docker-compose.prod.yml up --build
  ;;
*)
  docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --build
  ;;
esac

What are the caveats of this approach? (Haven't encountered any yet.)

I've done the same. Downside so far is to make sure not to type docker-compose up/down/etc without forgetting the correct additional docker-compose.yml. Perhaps compose knows not to apply commands to containers unless the same docker-compose files (used to start them) are specified?

[breuerfelix]

@umstek @dm17 https://github.com/docker/compose/issues/3729#issuecomment-623154878 <- this comment describes it the best

[dfanica]

So I'm guessing there's not way to remove an option as of yet despite many people requesting this feature?! :sad_panda:

[dm17]

Asking for clarification on the compose override docs here; seems relevant here: https://github.com/docker/docker.github.io/issues/12845

[doman18]

I have 4 different environments. Im adding now 5th where i want to remove 'image' sections from my services. Removing it from base docker-compose.yml and adding it for all 4 environments by overrides is far from productive solution. Some override with remove option would be perfect solution for such case.

[gnat]

For removing entire services this seems to work in the latest versions of docker-compose as of this writing.

version: "3.9"
services:
  app:
    deploy:
      replicas: 0

No solution for volumes that I can see, though.

[gamesover]

Please keep override as is. If you want to extend the ports, well, just repeat the old ports in the .override.yml file

Now, I can only extend ports, rather than override ports

[malcolm061990]

Let me explain my use case. I have compose file with kafka, postgres, service[1, 2, 3, 4] services. service[1, 2, 3, 4] are equal except the command field but service[1, 2, 3] depend on kafka and postgres, service4 doesn't depend anything. service[1, 2, 3] are defined using extension fields with depends_on field. But I don't require to run "dependent" (i.e. NOT dependent) services for launching service4, so I want to override it with null, nothing or empty value. How can I do that?

[miaoz2001]

Hi

It might be just me, but I want to override the environment by removing it from the root file, then use env_file in the .dev.yml file. :) Is there a workaround?

[divmgl]

I've found another actual use case for this. VSCode devcontainers will override the main command in order to keep the container running:

# .devcontainer/docker-compose.yml
services:
  app:
    command: /bin/sh -c "while sleep 1000; do :; done"

This is to avoid situations where the running command crashes and brings down the container (and the VSCode environment with it).

However, the problem is that if you have a base docker-compose.yml that exposes ports, like this:

# docker-compose.yml
services:
  app:
    ports:
      - "8081:8081"

Well now these ports are bound to the overridden sleep command and not the processes ran in the container. VSCode provides us with a forwardPorts option which automatically listens to binds and forwards them to the host, but this fails to work properly if the port is already statically bound in docker-compose.yml.

The workaround we found is to launch the process bound to all interfaces (0.0.0.0).

However we could have saved a 💩 ton of time (and not modified our codebase to fit this contrived use-case) if we could do something like this in the overriding docker-compose.yml:

# .devcontainer/docker-compose.yml
services:
  app:
    ports: ~

Now we can let VSCode forward ports from the container without binding to all interfaces.

This is valid YAML syntax btw.

You'd be able to keep overrides working as-is while also adding this feature.

The logic would be to ignore entire keys if a ~ or a null is in the overriding file.

[jawys]

If I recall correctly, it did not bind the port to any sleep command while using dev containers? At least on my side 🤔

[MrChrisRodriguez]

I know this is about ports, and someone else was talking about volumes... in my case, I want to overwrite profiles, another complex type (list).

I echo other people's sentiments:

• to me, at least, override implies replace, not merge, and especially not "replace in some cases and merge in others"
• there should be a way to distinguish between merge and overwrite behaviors because both are useful

[guvra]

I have another use case: I want to replace an image declared in docker-compose.yml with a build declared in docker-compose.override.yml.

docker-compose.yml:

myservice:
    image: xxxx

docker-compose.override.yml :

myservice
    build: ./docker/myservice

What I expected: Docker Compose builds my dockerfile and completely ignores the image tag declared in docker-compose.yml.

What happens: Docker Compose builds my dockerfile and tags it with the image name declared in docker-compose.yml.

---

edit: "solved" by overriding the image tag and setting it to {projectname}{service_name} (default value used by compose when no image tag is specified):

myservice:
    image: ${COMPOSE_PROJECT_NAME}_myservice
    build: ./docker/myservice

[ahmafi]

The word override is kinda misleading. With the current behavior of not supporting real overriding (removing and setting something else), I'm using 3 docker files:

docker-compose.yaml : base configs that are common between production and development environments. docker-compose.dev.yaml : development environment specifc config. docker-compose.prod.yaml: production environment specifc config.

This is my use case, and ofc there might be other use cases that can't work without a real override option.

[REDLINK]

For removing entire services this seems to work in the latest versions of docker-compose as of this writing.

version: "3.9"
services:
  app:
    deploy:
      replicas: 0

Interestingly, the documentation explicitly states that this

only takes effect when deploying to a swarm with docker stack deploy, and is ignored by docker-compose up

Using profiles could be an alternative:

services:
  app:
    profiles:
       - disabled

Where "disabled" is just an arbitrary name for a profile that's never enabled. Obviously, that doesn't help if you're already using profiles and would need to remove one or all of them. Anyway, an advantage would be that it can be overridden from the commandline to selectively start a service.

[kj]

Perhaps there could be something along the lines of omit_keys (for excluding) and pick_keys key (for including only those specified; in case of large configurations) allowed at any level of nesting in the YAML which applies to any previously specified YAML. Then you could remove services (in this case a 'foo' service) with:

services:
  omit_keys:
    - 'foo'
  bar:
    # ...

You could override ports by removing that key from a service:

services:
  foo:
    omit_keys:
      - 'ports'
    ports:
      - '8080:8080' # now the only port specified

Being usable at any level of nesting would make this very flexible, so you could remove or override just about anything (service, ports, volumes, any arbitrary key), while retaining the merging behaviour where it is desirable.

I haven't thought about this too deeply, so I'm not sure whether it's practical.

[logopk]

@kj sounds reasonable without problems with backward compatibility

[osmarcf]

There were several good suggestions about how to handle override. Let me suggest something also (trying to cover some of the cases I've seen reported in this thread):

Considering the default behavior of override is to merge properties. But, if specified, a prefix sign would change the default override behavior:

• a + sign forces merge with previous contents of the referred property;
• a ~ sign forces replace a previously referred property;
• a ! sign forces remove a previously referred property.

Unfortunately it does not respect the file pattern backwards compatibility, but it does not change the default behavior.

[signalwerk]

@osmarcf I think I don't understand 100% your suggested taxonomy. You speak about prefixing the key. right?

Something like that:

services:
  foo:
    "!ports":
      - '8080:8080'

Please forgive me if I misunderstood your suggestion.

Why would we need the +? Merging is the default pattern. No?

[osmarcf]

Yes, @signalwerk !

Sorry that I haven't provided an example, but you understood it.

Considering (order matters): docker compose -f first.yaml -f second.yaml

The + sign would serve as an "explicit" merge. Indeed, leaving a key without it would have the same behavior (merge).

Using + sign (default behavior): first.yaml:

services:
  foo:
    ports:
      - '8080:8080'

second:yaml

services:
  foo:
    ports:
      - '9090:9090'

resulting yaml:

services:
  foo:
    ports:
      - '8080:8080'
      - '9090:9090'

---

The ~ sign would replace any key set in a previous file.

first.yaml:

services:
  foo:
    ports:
      - '8080:8080'

second:yaml

services:
  foo:
    ~ports:
      - '9090:9090'

resulting yaml (replace any previous key):

services:
  foo:
    ~ports:
      - '9090:9090'

---

The ! sign would remove key set in any previous file. first.yaml:

services:
  foo:
    ports:
      - '8080:8080'

second:yaml

services:
  foo:
    !ports:
      - '8080:8080'

resulting yaml (remove key):

services:
  foo:

---

Maybe with these examples I could clarify what I thought.

[signalwerk]

@osmarcf thank you for answering my question and providing the great examples.

[M-Pixel]

Best practice is to not include anything in docker-compose.yml that you might not want to have in another environment.

This is certainly the way to go for most software.

When contributing to an open-source project, however, that usually isn't a practical option.

[hubertnnn]

Best practice is to not include anything in docker-compose.yml that you might not want to have in another environment.

This is certainly the way to go for most software.

When contributing to an open-source project, however, that usually isn't a practical option.

No its not. Most software would use sensible defaults and allow overwriting them with custom values. A database will use /var/lib/mysql as its data folder by default if you don't explicitly set it. A web server would listen on port 80 unless you explicitly change it. An image converter would use 90% quality setting on jpeg unless explicitly set to something else.

[4n70w4]

Hi! Any news?

[jawys]

bump

[OpNop]

For me the big issues is trying to remember syntax to use other compose files, what would be nice is if the docker compose up command had an easy way to swap the override file

example would be

docker compose up runs like normal using the optional docker-compose.override.yml

docker compose -o dev up runs docker compose -f docker-compose.yml -f docker-compose.dev.yml up

docker compose -o prod up runs docker compose -f docker-compose.yml -f docker-compose.prod.yml up

Basically a flag to change the override file name

[InCogNiTo124]

@OpNop how about writing a dev.env and a prod.env file, with respective COMPOSE_FILE=... entries and them just running docker compose --env-file=prod.env up

[CRC-Mismatch]

Best practice is to not include anything in docker-compose.yml that you might not want to have in another environment.

This is certainly the way to go for most software. When contributing to an open-source project, however, that usually isn't a practical option.

No its not. Most software would use sensible defaults and allow overwriting them with custom values. A database will use /var/lib/mysql as its data folder by default if you don't explicitly set it. A web server would listen on port 80 unless you explicitly change it. An image converter would use 90% quality setting on jpeg unless explicitly set to something else.

TL;DR: IMHO, at least the YAML ~ should be used as an "eliminate everything under here" as it's logically supposed to be (null is not "empty array|string|whatever", and an explicit ~ is an explicit, intentional null, akin to "there should be nothing here" or "this is nothing").

@hubertnnn I'm unsure if you disagree with "certainly the way to go for most software" or "however, that usually isn't a practical option", but... I'm giving my 2c anyway :shrug:

Sometimes, in a corporate team the "sensible default" may be to expose the service in its' default port, since at any given time, some random developer may need to clone the source and quickly reproduce and debug some failures that are happening in production. The real world isn't quite "sensible" as we all would hope it to be...

Sometimes, things can (and do) go wrong in production, and whoever is available will have to fix it ASAP. So, to make things easy, the projects are all created with all dependency services exposed at their default ports, and important non-environment-specific IDE configurations are also committed. With this, even developers that don't have much experience (if any at all) with Docker and Docker Composer can just clone the project, open it up in the IDE, "press play" (or follow some foolproof 2-step instructions to bring the stack up) and debug whatever they need: send requests to APIs, access the back-offices in their browser, run queries directly in the databases or check whatever else they may need to in any service via local tools, without having to waste time finding out which ports lead to what and modify profiles or set up new connections to "make it work".

It may not seem like it, but this saves a lot of time, usually wasted by an inexperienced (in Docker) developer thinking he's done something wrong and trying all sorts of remedies, or just resorting to "whatever is working" before asking the team; usually flying blind and taking 5x longer to investigate a tiny problem or validate his new feature - and I've seen worse, some times, with no way to tell, the developer "sticks to what they CAN tell" and end up completely reinventing the wheel to solve a basic problem that could be easily noticed with direct access.

OTOH, if an experienced developer has to implement something that requires two projects running locally at the same time, he'll have to edit the committed docker-compose schema and risk committing something that wasn't supposed to be committed, instead of being able to just create a docker-compose.override.yml and override the "other" project's exposed ports with ports: ~...

To me, going with "That's how software is supposed to be developed" and "it's best practice to do it like so" are just bad excuses for either "we don't know how to make it work", "we don't want to spend the time and effort implementing it" or "it's good enough as it is, so we don't think it's needed". There are many different workflows, dynamics, paradigms and philosophies when working with software development, and there's no "one fits all" methodology when it comes to DevOps...

That being said, I'd understand if Docker came out and explained their positions on why they won't support it, or if they will, but it isn't in their foreseeable milestones; being open-source, anyone can fork and open pull requests, so that's just fine, because someone else may have enough time to spare, coupled with the expertise and a firm resolve to contribute it. But if they're mute, or just throwing excuses around for 7 years now, it makes it seem like they don't care at all :man_shrugging:

Also, any "backwards compatibility" argument falls flat since there's a version property. If backwards-compatibility is to be kept between all versions, then what's the use of having that? And even worse, with semantic versioning... IMO, even if it isn't required, "if there's no version property, consider it to be the latest 'LTS' version". The contents are to be interpreted by the rules of that specific version. With such, if there are BC problems with breaking changes, then the problems are not with what's being implemented, they're in how the version is considered... If it's only considered BEFORE checking all overrides files, then it's a misleading capability, and mistakes have been made; the last override file SHOULD have priority - and it's the overriding user's problem to sort all changes out from previous files in it. If the version is immutably read from the original file, the only easy way out would be to come up with new filenames for a new schema - and I'd gladly vote for that. And with it, probably have a comment header, or something else that isn't parsed with everything else in each file, in order to decide HOW to parse everything else for that file.

[gnat]

Yet another obvious solution Docker Inc. could use to solve this.

Currently, during overrides:

• Maps are replaced.
• Lists are appended.

Great, but, error: services.*.ports must be a list services.*.volumes must be a list

Minimum viable solution: Allow ports and volumes to be maps as well (like, environment etc)

Ideally though, the superior developer experience would be the ability to explicitly "clear a list" when overriding.

[ndeloof]

@gnat there's some special handling for ports and volume: those are internally converted into maps, using volume target and port {target,ip,published,protocol} as key so that merging overrides does not produce duplicates. This is not exposed in the compose.yaml file format to maintain backward compatibility, but still you should get an equivalent behavior as maps.

I've been investigating options we have to support https://github.com/compose-spec/compose-spec/issues/284 in compose-go. While not trivial, I expect we get some viable solution ASAP

[gorbak25]

Also came across this .-.

[debbyglance]

Just chiming in to say "me too"

[ndeloof]

https://github.com/compose-spec/compose-spec/pull/340 has been approved and merged support in compose-go (https://github.com/compose-spec/compose-go/pull/380) will be available in next release

[J7mbo]

Damn, I remember reading this in 2016 and 7 years later it's still nice to know that it got in.

[default-value-serhiia]

Yeah! It finally happened! Thank you!

[hubertnnn]

I feel this is still not enough.
Spec mentions the ability to remove elements, not overwrite them. So based on new spec:

# docker-compose.yml
services:
  foo:
    ports:
      - 80:80
      - 443:443

docker-compose.override.yml
services:
  foo:
    ports: !reset

Will remove those ports. But there is still no way to replace ports with custom ones, unless this is going to work (which I doubt since its not a valid yaml):

docker-compose.override.yml
services:
  foo:
    ports: !reset
      - 8080:80
      - 8081:443

[ndeloof]

@hubertnnn yes indeed, !reset only can be used reset value to type's default (empty) to replace port, the simplest option for now is to rely on variables:

services:
  foo:
    ports:
      - ${HTTP_PORT:-80}:80
      - ${HTTPS_PORT:-443}:443

so you can run compose with a local .env file to set those variable to 8080,8081

[kj]

I think this is where I was confused too. What if you want to remove a port (or volume, or similar) rather than just replace it? I'm struggling to see how you can actually replace the whole value of ports.

[dokgu]

So how are we supposed to override ports using docker-compose.override.yml?

I have a docker-compose.yml file with a predefined port mappings - those ports are already in use so I need to override them. I don't want to make any modifications to the docker-compose.yml file because my changes will get overwritten with updates.

[olitomlinson]

yeah, I'm struggling removing all ports from a Service in the override. (I'm not intending to replace, just remove all ports)

In the override, if I do this :

version: '3.4'
services:
  workflow-app:
    ports: !reset 

I get the error : services.workflow-app.ports must be a list

So I try this :

version: '3.4'
services:
  workflow-app:
    ports: !reset 
      - ""

I get the error : error decoding 'ports': No port specified: <empty>

---

The only work around I've found is to create an entry for some non-existant ports, which is just a mess, like this :

version: '3.4'
services:
  workflow-app:
    ports: !reset 
    - "9999:9999"

What am I missing here? @ndeloof any ideas? TIA

[ndeloof]

services:
  workflow-app:
    ports: !reset [] // empty list won't trigger a parsing error

(and please, don't set a version: attribute, this is an obsolete practice!)

[olitomlinson]

@ndeloof awesome thanks so much! Thank you for the guidance around the obsolete practice. Much appreciated.

[kj]

@ndeloof I just want to check if I understand correctly.

ports: !reset []

Here, the value [] is ignored. It's just the default value for ports. So whatever you pass to ports is ignored, meaning that you cannot override it with a new set of ports. For example:

# docker-compose.yml
services:
  foo:
    ports:
      - '1000:1000'
      - '2000:2000'

# docker-compose.override.yml
services:
  foo:
    ports: !reset
      - '1000:1000'

The intention being to remove port 2000, but keep port 1000. That isn't possible at all currently (or anything involving replacing the full list)?

[ndeloof]

The intention being to remove port 2000, but keep port 1000. That isn't possible at all currently (or anything involving replacing the full list)?

yes indeed, this isn't yet supported. You will have to use 2 override files, first one to reset to empty list, second one to set port 1000 or maybe you can tweak the original compose file to offer more flexibility via variables ?

[kj]

Thanks for clarifying @ndeloof (and so quickly!). It's good at least to know for certain that I'm not missing something,

[douardda]

Hi @ndeloof

yes indeed, this isn't yet supported.

Does it mean you plan to support it in the future? (cannot find an issue for this but I may have missed it).

[ndeloof]

feel free to create an issue on github.com/compose-spec/compose-go for this purpose I don't have any specific plan for this, but we are debating a major refactoring of compose-go library which could make this easier in the future

[quarky42]

My development system has https_proxy and http_proxy environment variables set in order for applications (python, npm) to download their dependencies. Docker Desktop assumes that if I have a proxy defined on the system, come hell or high water it is going to force my containers to have that proxy in them. On Linux I can add the proxy to the /etc/systemd/system/docker.d/proxy.conf file and Docker will respect the fact that I don't want that environment variable stuffed into my containers. I haven't found a similar method for Windows yet. It knows about the proxy so I've got to deal with it showing up in the container.

I need the proxy to be used for build time, and I'm careful to use "ARG" in my Dockerfiles for building so that I can use the proxy during build time and exclude it for runtime, but docker compose has other plans.

Being able to: environment: !reset - VAR1: var1_value_here and not get any other variables besides VAR1 in my container is exactly what I want. Others that want to inherit environment variables from the host can simply leave off the !reset.

I tried using - http_proxy: ~ but I can see from the discussion above that docker compose doesn't implement that yaml specification correctly. Explicitly null > implicitly pulled from the host environment.

Being able to explicitly tell docker to let me have 100% control of what environment variables get defined in the docker-compose.yml would cut back on a lot of overhead dealing with corporate environments.

If someone has created the issue over on the compose-spec/compose-go github, can you please link it here as I would like to add my support for this feature. I have been fighting environment variable proxy garbage off and on for a year and I'm sick of it. I need a solution, like the one proposed above, that works on all OSes regardless of the underlying config. At the docker-compose.yml and docker-compose.override.yml level I ought to be able to definitively nail down the "truth" of how a container is defined and have it be defined that way on any system I run it.

[ndeloof]

@quarky42 HTTP_PROXY (and related) build args are automatically set based on your client configuration (~/.docker/config, proxies section`), you don't need those to be declared in your compose file.

[quarky42]

@quarky42 HTTP_PROXY (and related) build args are automatically set based on your client configuration (~/.docker/config, proxies section`), you don't need those to be declared in your compose file.

I don't want them to be set. I don't have them set in ~/.docker/config because that will set them in the containers.... Which is what I don't want.

On Linux, the solution is to set the environment variables in /etc/systemd/system/dockerd/proxy.conf

This works. There is no equivalent for Windows. Having !reset at a block level would ensure that I can create a config that explicitly denies all environment variables. When working on a project shared by multiple developers, it is very important to be able to have a consistent container generation.

I want to EXCLUDE all (100%) environment variables at the docker-compose.yml level. Currently, proxy values are being carried into my containers. Having support for !reset on a block level would ensure that only explicitly set values were used.