Closed vochicong closed 6 years ago
Hi @vochicong
Thank you for your report. We don't consider this to be a bug. Is this causing issues with your application?
Hi @shin- . Yes, I have dev
built upon prod
upon base
.
#Dockerfile.base
FROM ruby
...
#Dockerfile.prod
FROM myapp-base
...
#Dockerfile.dev
FROM myapp-prod
...
#docker-compose.yml
version: '2'
services:
base:
image: myapp-base
build:
context: .
dockerfile: Dockerfile.base
prod:
extends: base
image: myapp-prod
build:
context: .
dockerfile: Dockerfile.prod
dev:
extends: prod
image: myapp-dev
build:
context: .
dockerfile: Dockerfile.dev
Ok, one simple solution would be to run docker-compose build base && docker-compose build prod && docker-compose build dev
instead, but if you have some time, I highly recommend you look into using multi-stage builds: https://docs.docker.com/engine/userguide/eng-image/multistage-build/
@shin- I would like to re-open this, I am following multi stage builds that depends on another multi stage builds (micorservices). I would like the docker compose to build in the order specified , so that we build our common base first, then depends all our services on top of this. If the common base image exists already, compose skips it even though the Dockerfile for the base image have been changed, and the dependent images use the old image, literally wanting me to run compose twice. (BTW i dont want to use a script that builds it as compose definition is super easy to pass metadata as env variables during build).
Users would expect that docker compose builds images in the order specified. Multi-stage build is for building an image efficiently.
I run into the same problem but solved by reordering the service definitions in docker-compose.yaml file. docker-compose follows the service order defined in that file. See: https://github.com/docker/compose/blob/e9220f45df07c1f884d5d496507778d2cd4a1687/compose/project.py#L182-L183
For me, the ordering in the docker-compose.yaml file worked until a recent update to docker. It now builds the services in parallel (See the default value to the --parallel
option here) Any efforts to serialize the build failed (i.e. docker compose --profile production build --parallel=false
) Besides, the --parallel
flag has been deprecated.
I'll have to rely on a script to build each service individually. 😞
Multi-stage build is probably the best and most correct solution, but for some, it isn't quite enough. For me, the reasons are as follows:
Seconding what @grahamaj said. I have a compose file with three services (A, B, and C). All have separate Dockerfiles in their own directories (./a_dir
, ./b_dir
, and ./c_dir
). But B's Dockerfile starts with:
FROM A
and C contains a line like:
COPY --from=A /some/directory /some/directory
Meanwhile, docker-compose.yml
looks like:
services:
A:
build:
context: ./a_dir
B:
build:
context: ./b_dir
C:
build:
context: ./c_dir
Now, suppose that I make a change to A's Dockerfile which modifies /some/directory
. If I run docker compose build
, A's image should be built prior B and C, otherwise C might use the old version of /some/directory
and B might extend the wrong version of A's image. If the ordering of the services in docker-compose.yml
were respected, the behavior would be correct. However, in my testing, Docker Compose does not build the images in the order specified.
I originally used a multi-stage file with target
s set as appropriate for this, but that approach opens up an entirely new can of worms. Namely, because B and C are not dependent on each other, and were built as services in their own right, Buildkit/Compose would try to build them in parallel and would (bizarrely) often duplicate the execution of RUN
commands in previous stages in the process. This made builds take far longer than they should. That was what lead me to abandon the multistage Dockerfile approach and move to separate contexts for each. After all, this really isn't what multi-stage builds are for.
So, I can of course build them in order manually. But my scripts which used to be able to simply use docker-compose build
have to be changed. Not a big deal, but it really does seem like Docker Compose should either be able to resolve these dependencies or allow a build ordering to be explicitly specified.
A lot of threads on StackOverflow as well as issues in this repo seem to be under the impression that depends_on
solves this issue, but it does not, it only affects the order in which each service is started at run time. Some analogous option for build-time, though, would be quite helpful.
I would be happy with a build_depends_on
option that allows to specify dependencies.
Any update? I'm having a similar issue https://stackoverflow.com/questions/73677392/how-to-force-docker-compose-to-wait-for-one-build-to-complete-before-building-an
@shin-, could we reopen the ticket? It still exists.
Seconded. Having the same problem, and the ability to force docker NOT to parallelize certain build steps would be very useful.
(And also seems like a requirement for sanely implementing parallelism-by-default in the first place.)
Same issue where I have 2 services built using their own Dockerfile
which starts by a FROM:my-shared-image
.
The my-shared-image
image is built by a third, dedicated Docker Compose on which the other 2 depends (depends_on
).
At first implementation it worked on Docker Compose v1, then, with v2, I had issues with Docker trying to pull image my-shared-image
from the public registry (docker.io/library/my-shared-image
) which I fixed with pull_policy: never
(or I think I had).
docker-compose.yml
:
services:
# Special service to build an image (named "my-shared-image") once for
# other services to use.
_shared_builder:
build:
context: .
dockerfile: ./my-shared-image.Dockerfile
command: "echo Build is done"
image: "my-shared-image"
pull_policy: never
restart: "no"
service-a:
build:
args:
- DOCKER_BASE_IMAGE=my-shared-image
context: .
dockerfile: ./service-a.Dockerfile
depends_on:
- _shared_builder
pull_policy: never
service-b:
build:
args:
- DOCKER_BASE_IMAGE=my-shared-image
context: .
dockerfile: ./service-b.Dockerfile
depends_on:
- _shared_builder
pull_policy: never
service-a.Dockerfile
:
ARG DOCKER_BASE_IMAGE
FROM ${DOCKER_BASE_IMAGE}
RUN echo "I am service A."
service-b.Dockerfile
:
ARG DOCKER_BASE_IMAGE
FROM ${DOCKER_BASE_IMAGE}
RUN echo "I am service B."
Running docker-compose up -d --force-recreate --build
fails, but the following works:
docker-compose build --no-cache _shared_builder && \
docker-compose up -d --force-recreate --build
This issue is quite old, but I would like to toss my hat in favor of some mechanism of build dependencies. My proposal is a depends
key for the build
clause which contains a list of build directives, perhaps subsets of the build
schema, which must be completed before the main image can be built. With yaml's indirection features, this could be made to be quite efficient from a developer standpoint while being a non-breaking addition to the spec.
I would be happy with a
build_depends_on
option that allows to specify dependencies.
Please reopen this, proposal from above makes sense and would help a lot.
I hope someone breaks your dependencies half as bad as you have broke my entire build chain. The work-around proposed below is absolutely asinine. I'd have to manually write 30-40 build commands, instead of giving users the ability to manually set a build order.
I'd be fired for something this short-sighted.
Ok, one simple solution would be to run
docker-compose build base && docker-compose build prod && docker-compose build dev
instead, but if you have some time, I highly recommend you look into using multi-stage builds: https://docs.docker.com/engine/userguide/eng-image/multistage-build/
build_depends_on
I believe this is a good solution, I'm trying to build a container based on another and docker always ignores despens_on and builds in the wrong order.
docker-compose build does not follow the order in which the services are specified.
will actually try to build in the order
base
,dev
andprod
, which is alphabetical order and not the order specified on the command line.