docker-compose build <service-name> not following depends_on order

[ArtyMcFlyHA]

Description of the issue

docker-compose build <service-name> does not appear to follow depends_on like docker-compose up -d <service-name>. Note, this is different than https://github.com/docker/compose/issues/5228 which has a similar wording, but the linked issue does not use depends_on in the docker-compose.yml file.

Context information (for bug reports)

The Dockerfiles in question:

FROM alpine:3.8
RUN apk update && apk upgrade
WORKDIR /home

FROM my-private-registry/alpine:3.8
COPY npmrc /home/.npmrc
RUN apk add nodejs-current && node -v
WORKDIR /home

Output of docker-compose version

docker-compose version 1.22.0, build f46880f

Output of docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     true

Output of docker-compose config (Make sure to add the relevant -f and other flags)

services:
  alpine:
    build:
      context: /Users/aweborg/Documents/workspace/Utilities/dockerImages/alpine
    image: my-private-registry/alpine:3.8
  nodejs:
    build:
      context: /Users/aweborg/Documents/workspace/Utilities/dockerImages/nodejs
    depends_on:
    - alpine
    image: my-private-registry/nodejs:9.11
version: '3.0'

Steps to reproduce the issue

1. (empty registry for both images) run docker system prune -a (to ensure no local images cached)
2. run docker-compose build nodejs

Observed result

nodejs is attempted to be built without attempting to build alpine first and an error occurs as a result. (stacktrace below)

Expected result

I expected alpine to build first, nodejs to build second. If docker-compose up -d nodejs is run instead of docker-compose build nodejs than alpine is built first as expected. I am expecting the following output:

Building alpine
Step 1/3 : FROM alpine:3.8
3.8: Pulling from library/alpine
4fe2ade4980c: Pull complete
Digest: sha256:621c2f39f8133acb8e64023a94dbdf0d5ca81896102b9e57c0dc184cadaf5528
Status: Downloaded newer image for alpine:3.8
 ---> 196d12cf6ab1
Step 2/3 : RUN apk update && apk upgrade
 ---> Running in 8cc45ac07394
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
v3.8.1-59-ge104e0f441 [http://dl-cdn.alpinelinux.org/alpine/v3.8/main]
v3.8.1-42-ge6bc061baf [http://dl-cdn.alpinelinux.org/alpine/v3.8/community]
OK: 9546 distinct packages available
OK: 4 MiB in 13 packages
Removing intermediate container 8cc45ac07394
 ---> 9fb308b80a88
Step 3/3 : WORKDIR /home
 ---> Running in 55e07020306e
Removing intermediate container 55e07020306e
 ---> 1d0b0417c6de
Successfully built 1d0b0417c6de
Successfully tagged my-private-registry/alpine:3.8
Building nodejs
Step 1/4 : FROM my-private-registry/alpine:3.8
 ---> 1d0b0417c6de
Step 2/4 : COPY npmrc /home/.npmrc
 ---> 7ea90e6de2a0
Step 3/4 : RUN apk add nodejs-current && node -v
 ---> Running in 740846cfb3fa
(1/9) Installing ca-certificates (20171114-r3)
(2/9) Installing c-ares (1.14.0-r0)
(3/9) Installing libcrypto1.0 (1.0.2p-r0)
(4/9) Installing libgcc (6.4.0-r9)
(5/9) Installing http-parser (2.8.1-r0)
(6/9) Installing libssl1.0 (1.0.2p-r0)
(7/9) Installing libstdc++ (6.4.0-r9)
(8/9) Installing libuv (1.20.2-r0)
(9/9) Installing nodejs-current (9.11.1-r2)
Executing busybox-1.28.4-r1.trigger
Executing ca-certificates-20171114-r3.trigger
OK: 31 MiB in 22 packages
v9.11.1
Removing intermediate container 740846cfb3fa
 ---> ab51418bf940
Step 4/4 : WORKDIR /home
 ---> Running in ee83f6b41566
Removing intermediate container ee83f6b41566
 ---> d1d7a9a120ba
Successfully built d1d7a9a120ba
Successfully tagged my-private-registry/nodejs:9.11

Stacktrace / full error message

> docker-compose build nodejs
Building nodejs
Step 1/4 : FROM my-private-registry/alpine:3.8
ERROR: Service 'nodejs' failed to build: manifest for my-private-registry/alpine:3.8 not found

Additional information

macOS High Sierra version 10.13.6

Note, docker-compose build appears to work as expected (though that may be due to alphabetical ordering of service names?). However, the above is a boiled down least code replication of the issue. A more practical example of where this issue becomes a problem is when there are several dozen services and only 1 or 2 need to explicitly have their images/dependencies built.

Also note, I replaced the private registry domain I'm using with my-private-registry. If you use a made up repository, the error will read Service 'nodejs' failed to build: pull access denied for my-private-registry/alpine, repository does not exist or may require 'docker login' instead of ... failed to build: ... not found. That said, expected output should be the same as above.

[shin-]

Hi @ArtyMcFlyHA

The effects of the depends_on key are clearly outlined in the documentation. As you can see, build ordering is not an intended usage of depends_on. More generally, higher-level build automation is not an intended usage of Compose, and we recommend finding complementary tools to fulfill this usage (dobi is usually a good candidate)

[ArtyMcFlyHA]

Hi @shin-

Thanks for the quick response as well as the clarity you've provided! I will definitely look into dobi, thank you for the recommendation! :)

I will say though, I believe as written at the time of this issue creation, we see the documentation painting different pictures. IMHO, it does not clearly outline that build ordering is not an intended usage. https://docs.docker.com/compose/compose-file/#depends_on the "simple example" even includes a service that has a build context.

I realize documentation is a separate issue, but I'd advise that "simple example" have it's build context removed and There are several things to be aware of when using depends_on have the case of docker-compose build be added to it.

I'm not alone in thinking this should've worked, I tracked down a couple posts on other internet forums on the topic where other devs also believe that depends_on would dictate build order. I'm going to link your response to said third party forums.

[codenjoyme]

This solution can help

[tswaters]

There is nothing in the docs about what the effects of depends_on are in the context of docker build. The way it's written, docker build is not even mentioned - to me it's not immediately clear if this is an omission, or a lack of support.

Further, docker build does respect depends_on while docker build <service> does not. Maybe it's a "feature" of docker build, or maybe docker build <service> has a bug. Nothing is mentioned one way or the other.

[lhuhn-wish]

The effects of the depends_on key are clearly outlined in the documentation. As you can see, build ordering is not an intended usage of depends_on. More generally, higher-level build automation is not an intended usage of Compose, and we recommend finding complementary tools to fulfill this usage (dobi is usually a good candidate)

@shin- Is dobi still recommended? I see that dobi has not received commits since July. I am trying to evaluate what tool to use because we have outgrown docker-compose build.

[rayzorben]

I am here because like others I feel the documentation is lacking. I spent a couple hours because of this simple example on https://docs.docker.com/compose/compose-file/

version: "3.7"
services:
  web:
    build: .
    depends_on:
      - db
      - redis
  redis:
    image: redis
  db:
    image: postgres

There are no comments that lead me to believe that the behavior is anything but

1. create network
2. start db
3. start redis
4. build web - since db and redis should be running, the network should be available

[jgonzalezd]

Wow! So build won't build the services in which it depends_on like up. That's a bit of a suprise. Good to know. I'm glad I found this thread.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.

[Neirda24]

Hi, I'd like to re-open this issue as I think having a way to prioritize docker builds would be very helpful. Sometimes we have a docker-compose file where we build several images. One "core" (nginx for example) and others that will extend it by adding some specific configurations (php for example). When doing so, one has to be built before others. For quite some time (several years) docker-compose build was respecting that order. Suddenly when switching to parallel builds it does not anymore. I understand it wasn't the intend at first but has been widely used... Would it be possible to consider it again ?

Thanks.

[strophy]

I think it is now possible to prioritize builds correctly using the buildx bake command, as of buildx 0.8.x. See the "Create Build Pipelines by Linking bake Targets" section in this blog post: https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/

This does not seem to be supported in compose syntax, it is necessary to use the json/hcl bake syntax instead.