[BUG] Got services.NAME conflicts with imported resource after update from 2.21.0 to 2.24.1

ishkulov commented 8 months ago

Description

I have configuration based on this example https://docs.docker.com/compose/multiple-compose-files/include/#example. There three files:

# postgresql.yml
version: '3'
services:
  postgres:
    image: postgres

# compose.yml
version: '3'
include:
  - postgresql.yml

# compose.override.yml
version: '3'
services:
  postgres:
    ports:
      - 5432:5432

With docker compose v2.24.1, it shows error:

$ docker compose version
Docker Compose version v2.24.1
$ docker compose config
services.postgres conflicts with imported resource

But it works well with v2.21.0:

$ docker compose version
Docker Compose version v2.21.0
$ docker compose config
name: compose
services:
  postgres:
    image: postgres
    networks:
      default: null
    ports:
    - mode: ingress
      target: 5432
      published: "5432"
      protocol: tcp
networks:
  default:
    name: compose_default

Steps To Reproduce

Create directory with files above and run docker compose up (or config)

Compose Version

$ docker compose version
Docker Compose version v2.24.1

Docker Environment

$ docker info
Client: Docker Engine - Community
 Version:    25.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 682
 Server Version: 25.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-17-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.48GiB
 Name: ishkulov
 ID: V27O:S3O2:3SYQ:VHRF:HYQB:4CBG:APXI:JHXG:HFAX:CR4P:K5TG:AHMX
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

mkraft commented 8 months ago

I'm also experiencing this bug after upgrading Docker Desktop for macOS to v4.27.0. Also merging files.

ndeloof commented 8 months ago

The pipeline to parse, include and override compose yaml files has been redesigned with latest release. include is applied as last step in this process, the logic with this attribute is that you rely on a compose file designed by third party, as an atomic building block. The included resources are not supposed to be "patched" by your own compose files. Still, you can define imported resources with an override to match your use-case:

include:
  - path: 
      - postgresql.yml
      - postgresql.override.yml

you can then use this postgresql.override.yaml file to tweak the service being imported in your application

ishkulov commented 8 months ago

So, if I have 10 services included, I have to create 10 separate override files to change ports. That's sad

ndeloof commented 8 months ago

@ishkulov that's indeed a usability issue, need to think about it. Anyway I could ask : why isn't the included compose file usable without an override ? The main idea for this feature was to allow use of "reusable building blocks" that you can plug in your application without having to worry about their design (maybe a single container, maybe a dozen)

ishkulov commented 8 months ago

Anyway I could ask : why isn't the included compose file usable without an override ?

We use a compose.yml file for local development, as it describes all the services required for the project. Developers can modify ports in the compose.override.yml file as needed for local development and to prevent conflicts. Additionally, we include services in other files for environment-specific configurations, such as compose.testing.yml or compose.review.yml, as "reusable block" but not all of them.

patrickli commented 8 months ago

This completely broke my setup as well. I have a bunch of services defined in their own compose files where they can run individually. I also have another compose file that includes all of them and apply some overrides so some of the services can start to talk to each other.

Now even if I include the override file as the last include it still doesn't work with service.XXXX conflicts with imported resource message.

More importantly, why is this change not mentioned anywhere in your changelog? This is a major behaviour change that is clearly not backwards compatible.

ndeloof commented 8 months ago

More importantly, why is this change not mentioned anywhere in your changelog? This is a major behaviour change that is clearly not backwards compatible.

This is not documented as ability to override an included resource was not an expected feature - this only works as the legacy compose file parser allowed to, and it was redesigned in compose-go/v2. The intent of this feature is to be equivalent to coding language include|import|require dependency management: your "code" uses third party components, and those are (in most case) close to extension.

Your use case to reuse an existing component and tweak it's definition is already covered by extends. Any reason you can't rely on it ?

patrickli commented 8 months ago

I don't remember how exactly I ended up using include instead of extends, but that's the set up I have been using for a year or so. Given the comments by other users there are clearly other similar use-cases as I have. This is just a bit frustrating from an end-user perspective. If an upstream change is going break my setup then I would expect this to be communicated clearly, at the very least.

ndeloof commented 8 months ago

I fully understand your frustration with this unexpected change, not sure yet how we can manage this. This is the first time such a usage has been reported since we introduced include, if we want this to be supported we would need to add such a scenario to the test suite and get this fully documented.

leocencetti commented 8 months ago

The intent of this feature is to be equivalent to coding language include|import|require dependency management: your "code" uses third party components, and those are (in most case) close to extension.

This is only partially true. An include|import|require statement in code will also recursively resolve all the includes|imports|requires of the dependency, BEFORE moving to the next dependency.

If I specify a sequence of compose files, let's say a.yaml, b.yaml, and c.yaml, I expect (using the "code" analogy) each to be fully resolved in order: first a and its dependencies, second b and its dependencies, and finally c and its dependencies.

Also, the docs state:

include makes it easier to modularize complex applications into sub-Compose files. This allows application configurations to be made simpler and more explicit. This also helps to reflect in the config file organization the engineering team responsible for the code.

This is a common usage. The new behavior denies the possibility to tweak the default configuration (e.g. test a new image, add a network, change a port, etc) without modifying the original compose files.

ndeloof commented 8 months ago

The analogy I used here was not to dictate a behavior, but to explain where include comes from. We need to investigate how we can provide an explicit (i.e. documented and long-terms supported) way to tweak an imported model

hdepalma commented 8 months ago

We encounter the same problem and our project is now broken. We have a set of templatized applications all included in one compose file. Each included file can reference multiple services, this is why using extends was not very practical (with extends, each service has to be referenced). We also need developers to be able to override any service for development, debugging or testing purpose, such as mounting volumes. They cannot do it anymore with this change.

Happy to get some direction on how our project should be defined / organized with this new redesign.

michaelhscreencloud commented 8 months ago

The update unfortunately broke our setup as well.

Take the following example:

compose.yaml:

include:
  - services/test.yaml

services/test.yaml:

services:
  test:
    image: hello-world
    environment:
      BREAKING: "false"

Before the update, devs could define local overrides such as ports and envars in compose.override.yaml.

services:
  test:
    environment:
      BREAKING: "true"

After the update this results in the error: services.test conflicts with imported resource.

I don't see why this is an unusual use-case? And I can't find any mention in the docs of included services not being override-able.

include makes it easier to modularize complex applications into sub-Compose files. This allows application configurations to be made simpler and more explicit. This also helps to reflect in the config file organization the engineering team responsible for the code.

This statement also seems to support the use-case of breaking services into separate files as a means of organization.

What would be the "idiomatic" Docker Compose approach to achieve this?

ThomasTosik commented 8 months ago

This also breaks Visual Studio and Docker Tools.

On build

2>------ Rebuild All started: Project: docker-compose, Configuration: Debug Any CPU ------
2>docker-compose  -f "C:\Projects\OMITTED\docker\docker-compose.yml" -f "C:\Projects\GIT\OMITTED\docker\obj\Docker\docker-compose.vs.release.partial.g.yml" -p OMITTED --ansi never kill
2>services.someservice conflicts with imported resource
2>C:\Program Files\Microsoft Visual Studio\2022\Preview\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(518,5): warning : services.someservice conflicts with imported resource

On debug start:

1>------ Build started: Project: docker-compose, Configuration: Debug Any CPU ------
1>docker-compose  -f "C:\Projects\OMITTED\docker\docker-compose.yml" -f "C:\Projects\OMITTED\docker\obj\Docker\docker-compose.vs.debug.g.yml" -p helix --ansi never --profile "*" config
1>services.someservice conflicts with imported resource
1>C:\Program Files\Microsoft Visual Studio\2022\Preview\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(425,5): error DT1001: services.someservice conflicts with imported resource
1>Done building project "docker-compose.dcproj" -- FAILED.

ruscon commented 8 months ago

We also experiencing this issue after upgrading.

Our flow:

We have many microservices, each microservice has its own compose file with the main configuration (port, env, etc)
We have a separate mono repository that connects all microservices as git submodules. 2.1. Here we extend the compose file of each service and add general custom things like volumes and depends_on. 2.2. We have compose files, which include separate service's compose files into one, like a group of services. 2.3. docker-compose.yml includes groups of services (2.2.) that we need by default + env_file 2.4. docker-compose.override.yml - here we override the service using extends

Also, we use ${PWD} and I see that the path is being generated incorrectly. I see this if I try to run docker compose build from the root folder of the mono repository (found that it is because of build context with . value)

Everything worked before the update.

FFdhorkin commented 8 months ago

The pipeline to parse, include and override compose yaml files has been redesigned with latest release. include is applied as last step in this process, the logic with this attribute is that you rely on a compose file designed by third party, as an atomic building block. The included resources are not supposed to be "patched" by your own compose files. Still, you can define imported resources with an override to match your use-case:
include:
  - path: 
      - postgresql.yml
      - postgresql.override.yml
you can then use this postgresql.override.yaml file to tweak the service being imported in your application

This does not seem to work for me. My overrides won't work, even when I add the - path as shown and include as shown. I've tried several permutations, and continue to get conflicts with imported resource errors. As indicated by other posters, I have to split every single override into its own file now, even though the syntax is unchanged - i still need to have services: in these overrides.

Not sure how the Docker team anticipated people to use docker-compose.override.yml in the first place - why offer it and then deprecate it without warning? If it wasn't meant to override someone else's docker-compose file... why make it exist in the first place?

ndeloof commented 8 months ago

Not sure how the Docker team anticipated people ...

we didn't. Use of "local override for included resource" is a pattern we discovered with this issue - we have no way to guess how features are used in the wild. The regression isn't intentional (caused by a redesign of the legacy compose parser)

Aposhian commented 6 months ago

Also ran into this behavior and was confused. I anticipated that include: compose.yml would simply work as a shortcut for -f compose.yml. My opinion is that those two should work exactly the same. Or maybe there could be a different syntax for "include isolated" and "include with overrides". Maybe include namespaces or something?

One convenient thing about include is that it allows compose files to declare their own dependencies on other compose files (including override files before this change) whereas before I end up just having scripts that manage long lists of -f flags.

Essentially you could have a compose file like: "here's an additional service, and I also want to apply a standard set of overrides when I use this extra service." Using -f delegates to the caller of the compose command to get that right, but include can embed it. However, if include can't override, then you need to either:

make a completely separate compose target, and then create a new one with new services that "extend" all the previous ones. This is very cumbersome.
don't use include, and just expect the caller of the compose command to make sure that certain -f combinations are always used together.

luciangabor commented 6 months ago

or just include everything (whatever would be defined in the including file can also be included, and with this point of view the existing merge restriction is simply just an annoyance)

bblakely-anl commented 5 months ago

My use case is to allow a user to specify an environment variable for their local network subnet atop an included Suricata container that is otherwise configured. Like others, it's unwieldy (gitignore, Makefile, etc) to have multiple include files for different services/networks/etc, so would appreciate having this functionality restored. Thank you!

docker / compose