build not follow the order specified

[vochicong]

docker-compose build does not follow the order in which the services are specified.

docker-compose build base prod dev

will actually try to build in the order base, dev and prod, which is alphabetical order and not the order specified on the command line.

[shin-]

Hi @vochicong

Thank you for your report. We don't consider this to be a bug. Is this causing issues with your application?

[vochicong]

Hi @shin- . Yes, I have dev built upon prod upon base.

#Dockerfile.base
FROM ruby
...

#Dockerfile.prod
FROM myapp-base
...

#Dockerfile.dev
FROM myapp-prod
...

#docker-compose.yml
version: '2'

services:
  base:
    image: myapp-base
    build:
      context: .
      dockerfile: Dockerfile.base

  prod:
    extends: base
    image: myapp-prod
    build:
      context: .
      dockerfile: Dockerfile.prod

  dev:
    extends: prod
    image: myapp-dev
    build:
      context: .
      dockerfile: Dockerfile.dev

[shin-]

Ok, one simple solution would be to run docker-compose build base && docker-compose build prod && docker-compose build dev instead, but if you have some time, I highly recommend you look into using multi-stage builds: https://docs.docker.com/engine/userguide/eng-image/multistage-build/

[frezbo]

@shin- I would like to re-open this, I am following multi stage builds that depends on another multi stage builds (micorservices). I would like the docker compose to build in the order specified , so that we build our common base first, then depends all our services on top of this. If the common base image exists already, compose skips it even though the Dockerfile for the base image have been changed, and the dependent images use the old image, literally wanting me to run compose twice. (BTW i dont want to use a script that builds it as compose definition is super easy to pass metadata as env variables during build).

[vochicong]

Users would expect that docker compose builds images in the order specified. Multi-stage build is for building an image efficiently.

[cenkalti]

I run into the same problem but solved by reordering the service definitions in docker-compose.yaml file. docker-compose follows the service order defined in that file. See: https://github.com/docker/compose/blob/e9220f45df07c1f884d5d496507778d2cd4a1687/compose/project.py#L182-L183

[grahamaj]

For me, the ordering in the docker-compose.yaml file worked until a recent update to docker. It now builds the services in parallel (See the default value to the --parallel option here) Any efforts to serialize the build failed (i.e. docker compose --profile production build --parallel=false) Besides, the --parallel flag has been deprecated.

I'll have to rely on a script to build each service individually. 😞

Multi-stage build is probably the best and most correct solution, but for some, it isn't quite enough. For me, the reasons are as follows:

• Reuse of intermediate images: I have a build container that I make available in a development profile where developers can freely run the build, yet the build container is also used as an intermediate stage for the production image
• There is no way to name the images for the intermediate stages. They just end up with a random GUID (even for named stages). This is nearly meaningless to a human. Named images for intermediate stages help me:
  • debug at some stage, it is easy to determine which image to fire up
  • quickly identify where to look for problems when they arise
  • identify which images are no longer needed and can be pruned

[thomas-skillup]

Seconding what @grahamaj said. I have a compose file with three services (A, B, and C). All have separate Dockerfiles in their own directories (./a_dir, ./b_dir, and ./c_dir). But B's Dockerfile starts with:

FROM A

and C contains a line like:

COPY --from=A /some/directory /some/directory

Meanwhile, docker-compose.yml looks like:

services:
  A:
    build:
      context: ./a_dir
  B:
    build:
      context: ./b_dir
  C:
    build:
      context: ./c_dir

Now, suppose that I make a change to A's Dockerfile which modifies /some/directory. If I run docker compose build, A's image should be built prior B and C, otherwise C might use the old version of /some/directory and B might extend the wrong version of A's image. If the ordering of the services in docker-compose.yml were respected, the behavior would be correct. However, in my testing, Docker Compose does not build the images in the order specified.

I originally used a multi-stage file with targets set as appropriate for this, but that approach opens up an entirely new can of worms. Namely, because B and C are not dependent on each other, and were built as services in their own right, Buildkit/Compose would try to build them in parallel and would (bizarrely) often duplicate the execution of RUN commands in previous stages in the process. This made builds take far longer than they should. That was what lead me to abandon the multistage Dockerfile approach and move to separate contexts for each. After all, this really isn't what multi-stage builds are for.

So, I can of course build them in order manually. But my scripts which used to be able to simply use docker-compose build have to be changed. Not a big deal, but it really does seem like Docker Compose should either be able to resolve these dependencies or allow a build ordering to be explicitly specified.

A lot of threads on StackOverflow as well as issues in this repo seem to be under the impression that depends_on solves this issue, but it does not, it only affects the order in which each service is started at run time. Some analogous option for build-time, though, would be quite helpful.

[MattBussing]

I would be happy with a build_depends_on option that allows to specify dependencies.

[tran-simon]

Any update? I'm having a similar issue https://stackoverflow.com/questions/73677392/how-to-force-docker-compose-to-wait-for-one-build-to-complete-before-building-an

[AlexeyMoiseev]

@shin-, could we reopen the ticket? It still exists.

[GBirkel]

Seconded. Having the same problem, and the ability to force docker NOT to parallelize certain build steps would be very useful.

(And also seems like a requirement for sanely implementing parallelism-by-default in the first place.)

[C-Duv]

Same issue where I have 2 services built using their own Dockerfile which starts by a FROM:my-shared-image.

The my-shared-image image is built by a third, dedicated Docker Compose on which the other 2 depends (depends_on).

At first implementation it worked on Docker Compose v1, then, with v2, I had issues with Docker trying to pull image my-shared-image from the public registry (docker.io/library/my-shared-image) which I fixed with pull_policy: never (or I think I had).

docker-compose.yml :

services:

    # Special service to build an image (named "my-shared-image") once for
    # other services to use.
    _shared_builder:
        build:
            context: .
            dockerfile: ./my-shared-image.Dockerfile
        command: "echo Build is done"
        image: "my-shared-image"
        pull_policy: never
        restart: "no"

    service-a:
        build:
            args:
                - DOCKER_BASE_IMAGE=my-shared-image
            context: .
            dockerfile: ./service-a.Dockerfile
        depends_on:
            - _shared_builder
        pull_policy: never

    service-b:
        build:
            args:
                - DOCKER_BASE_IMAGE=my-shared-image
            context: .
            dockerfile: ./service-b.Dockerfile
        depends_on:
            - _shared_builder
        pull_policy: never

service-a.Dockerfile :

ARG DOCKER_BASE_IMAGE
FROM ${DOCKER_BASE_IMAGE}
RUN echo "I am service A."

service-b.Dockerfile :

ARG DOCKER_BASE_IMAGE
FROM ${DOCKER_BASE_IMAGE}
RUN echo "I am service B."

Running docker-compose up -d --force-recreate --build fails, but the following works:

docker-compose build --no-cache _shared_builder && \
docker-compose up -d --force-recreate --build

[rhelmot]

This issue is quite old, but I would like to toss my hat in favor of some mechanism of build dependencies. My proposal is a depends key for the build clause which contains a list of build directives, perhaps subsets of the build schema, which must be completed before the main image can be built. With yaml's indirection features, this could be made to be quite efficient from a developer standpoint while being a non-breaking addition to the spec.

[Getinge-Dumitru]

I would be happy with a build_depends_on option that allows to specify dependencies.

Please reopen this, proposal from above makes sense and would help a lot.

[hosein]

I hope someone breaks your dependencies half as bad as you have broke my entire build chain. The work-around proposed below is absolutely asinine. I'd have to manually write 30-40 build commands, instead of giving users the ability to manually set a build order.

I'd be fired for something this short-sighted.

Ok, one simple solution would be to run docker-compose build base && docker-compose build prod && docker-compose build dev instead, but if you have some time, I highly recommend you look into using multi-stage builds: https://docs.docker.com/engine/userguide/eng-image/multistage-build/

[luizoti]

build_depends_on

I believe this is a good solution, I'm trying to build a container based on another and docker always ignores despens_on and builds in the wrong order.