Closed slavdok closed 2 years ago
Any movements here ?
Why? I suppose that when it was added, named volumes didn't exist yet? Either way, why would anon volumes get this feature while named volumes wouldn't?
No, I think this was by design.
Anonymous volumes are generally considered "ephemeral" storage, so considered safe to remove/recreate. docker-compose
treats them a bit more "persistant" than docker run
; where regular docker run
-> docker rm
-> docker run
flow creates new anonymous volumes for each docker run
, Docker Compose tries to restore volumes on up
, even if the volume was anonymous. The --renew-named-volumes
option was added to make up
more closely match the docker run -> delete -> run
flow; because anonymous volumes don't have an easy-to-remember name, it would otherwise be more complicated to find the right volume(s) to delete.
Named volumes are considered to be used for persistent data; they're given a name because they usually are important (e.g. your database data), and the name can be used to reference them / find them back. They should not be deleted unless the user really wants to (which can be done on docker compose down --volumes
.
I understand there could be concurrency issues when using shared named volumes, but that could be mitigated when all other containers are mounting the volume as ro and only one does the (re)initialization.
Yes, provisioning the volume will be done by the first container that consumes it. If the volume is shared by multiple services, then the nocopy
option can be used to disable this behavior for containers that should not be used to provision the volume; https://docs.docker.com/compose/compose-file/compose-file-v3/#long-syntax-3.
Note that a volume cannot be deleted if it's still in use by a container, so deleting a volume that's used by multiple containers/services will always require those containers to also be re-created (deleted).
It's there a specific reason for your use-case to use a named volume? Perhaps volumes_from
would work for your use-case? https://github.com/compose-spec/compose-spec/blob/master/spec.md#volumes_from
My use case is the frontend / backend volumes binds here: https://github.com/openfoodfacts/openfoodfacts-server/blob/main/docker-compose.yml
Basically, we have a frontend container that builds NPM assets (css, js, node_modules) and a backend container that needs the output in read-only mode.
Currently, we have to run:
docker-compose down -v
docker-compose up -d
otherwise the volume content is not re-initialized when a new image is pulled. But this recreates all containers, not just the one that changed.
With the option, we could simply run:
docker-compose up -d --renew-named-volumes
when a new image is pulled.
@ocervell for this specific scenario you could use volumes_from: backend
in your frontend
service so that you don't need a named volume but just get the backend service to access files from frontend (anonymous) volume.
@ndeloof Can we use volumes_from
and change the target to a different directory ? We have a volume that's exported from the frontend
container:
volumes:
- product_images:/mnt/podata/product_images
and we need to bind it to a different path in the backend
container:
volumes:
- product_images:/opt/product-opener/html/images/products:ro
Any way to achieve this with anonymous volumes and volumes_from
?
nope, volumes_from require same bind mount target
looks like we need to --renew-named-volumes
option then :)
If these assets are static (i.e., the images are created at build-time of the image, and not generated/updated at runtime), perhaps something like https://github.com/moby/moby/issues/30449 would be useful for this scenario.
I'm (personally) a bit hesitant on adding a --renew-named-volumes
flag, for reasons listed above, and because it may not be granular enough (there's also an abundance of flags already; if this feature would be added, perhaps we should change the --force-recreate
to accept a list of object-types to recreate (services, volumes, ...).
But this recreates all containers, not just the one that changed.
With the current features provided, I think this scenario can be covered using profiles
; https://github.com/compose-spec/compose-spec/blob/master/spec.md#profiles
For example:
services:
one:
image: nginx:alpine
profiles: ["all-services"]
two:
image: nginx:alpine
profiles: ["all-services", "some-services"]
three:
image: nginx:alpine
profiles: ["all-services"]
four:
image: nginx:alpine
profiles: ["all-services", "some-services"]
Start all services
$ docker compose --profile=all-services up -d
[+] Running 5/5
⠿ Network compose_profile_default Created 0.1s
⠿ Container compose_profile_one_1 Started 1.4s
⠿ Container compose_profile_three_1 Started 1.8s
⠿ Container compose_profile_two_1 Started 1.5s
⠿ Container compose_profile_four_1 Started 1.8s
Verify that all services are running:
$ docker compose ps
NAME COMMAND SERVICE STATUS PORTS
compose_profile_four_1 "/docker-entrypoint.…" four running 80/tcp
compose_profile_one_1 "/docker-entrypoint.…" one running 80/tcp
compose_profile_three_1 "/docker-entrypoint.…" three running 80/tcp
compose_profile_two_1 "/docker-entrypoint.…" two running 80/tcp
Use the some-services
profile to remove/recreate some of the services
(note: it looks like there's a minor bug, where compose does not take into account services from other profiles in the stack)
$ docker compose --profile=some-services down -v
[+] Running 2/2
⠿ Container compose_profile_two_1 Removed 0.4s
⠿ Container compose_profile_four_1 Removed 0.4s
⠿ Network compose_profile_default Error 0.0s
failed to remove network 39cb990641f6b20313857d0b940c1f6d3bc1e22d773a7b5747f4728157a9104a: Error response from daemon: error while removing network: network compose_profile_default id 39cb990641f6b20313857d0b940c1f6d3bc1e22d773a7b5747f4728157a9104a has active endpoints
After this, only the services in the "some-services"
profile have been removed;
$ docker compose ps
NAME COMMAND SERVICE STATUS PORTS
compose_profile_one_1 "/docker-entrypoint.…" one running 80/tcp
compose_profile_three_1 "/docker-entrypoint.…" three running 80/tcp
Recreate the services to update them;
$ docker compose --profile=some-services up -d
[+] Running 2/2
⠿ Container compose_profile_four_1 Started 0.7s
⠿ Container compose_profile_two_1 Started 0.7s
And all services are up again;
$ docker compose ps
NAME COMMAND SERVICE STATUS PORTS
compose_profile_four_1 "/docker-entrypoint.…" four running 80/tcp
compose_profile_one_1 "/docker-entrypoint.…" one running 80/tcp
compose_profile_three_1 "/docker-entrypoint.…" three running 80/tcp
compose_profile_two_1 "/docker-entrypoint.…" two running 80/tcp
I like the idea --force-recreate
is extended to allow a list of resources to be recreated. Need to check how this can be combined with the boolean style it has today.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This issue has been automatically closed because it had not recent activity during the stale period.
I'm using a container-to-container top-level volume, cause php and nginx needs divide the same files (the static files are processed by php container).
And I really need to recreate these volume for production, otherwise I cannot copy the new files to these containers when rebuiding.
Using the two following commands works fine for me:
docker compose -p project-name down -v
docker compose up -d --build
The only problem is the gap time that containers stays stopped.
--renew-volumes
flag would be usefullAnother way would be a way to create the top-level volume as anonymous, so the current flag would work. See:
name: project-name
volumes:
volume_name:
create_as_anonymous:
services:
php:
volumes:
- volume_name
But this solution not seems cool. And this problem really has a business impact.
@Scemist you could use volumes_from
in your nginx service so that it get (shared) access to your php data (anonymous) volume.
if you want to use named volumes and get them renewed, then compose will anyway require to stop services, delete and recreate fresh new volumes (as name is unique, we can't "recreate" as with rolling updates), then restart the stack. This basically would be the same as running docker compose down -v && docker compose up
services:
nginx:
volumes_from:
- php
php:
volumes:
- /app
- another-volume:with-no-relation
Yes, this creates a anonymous shared volume, so I am able to start with the following command, and the volume are recreated:
docker compose -f docker-compose.prod.yml up -d --renew-anon-volumes --build
# Or the contracted form
docker compose up -Vd --build
volumes_from
approach:✅ The volume is anonymous, so can be recreated with --renew-anon-volumes
❌ Both containers will have the same directory, so differents paths like php-/app
:nginx-/var/www
isn't possible.
Docker version 20.10.21 Docker Compose version v2.13.0 Compose File Specification Version
indeed, volumes_from
don't let you customize the mount path. For your use case sounds easy to tweak nginx configuration accordingly.
Is your feature request related to a problem? Please describe. It is unnecessarily complicated to refresh/update the contents of named volume from container. When a named volume is created, it is initialized with the contents of directory from container. This behaviour is consistent between
docker
anddocker-compose
commands.But if the image is updated and it changes that initial content, the volume is not reinitialized. Using
docker
it's possible to manually delete that volume. Usingdocker-compose
, onlydown -v
would delete it, but that applies to all services. Thedown
command doesn't allow specifying a single service, and thestop
command doesn't have the-v
option.In a multi-service compose file, it is therefore outright impossible to reinitialize (or delete/recreate) the volume using
docker-compose
. Things only get more complicated when the named volume is shared by multiple services, as all containers must be removed (not just stopped) to delete/recreate the volume.Describe the solution you'd like Command
docker-compose up
has the following options:The option to recreate volume is already there, but it's limited to anonymous volumes only. Why? I suppose that when it was added, named volumes didn't exist yet? Either way, why would anon volumes get this feature while named volumes wouldn't?
I would like
--renew-named-volumes
option.I understand there could be concurrency issues when using shared named volumes, but that could be mitigated when all other containers are mounting the volume as
ro
and only one does the (re)initialization. Please give me the option, print a warning if you must, and let me decide if I want to risk concurrency and/or open handles issues.Describe alternatives you've considered One alternative I had working was a complex shell script that identified all running/stopped containers using a specific named volume, and then removing containers, removing volume and re-creating containers with reinitialized volume. The trouble was that if some containers using this named volume were created outside the current compose file (using
volumes external: true
), there was no way to bring them back without knowing what compose file launched them (or if they were launched bydocker
instead). Also, so much downtime across so many services wasn't ideal.Another solution that I am currently employing is keeping the image's volume-destined content outside of the actual volume, and then a needlessly large
entrypoint.sh
scriptrsyncs
from this "backup" location on container into the volume location, based oncmd
that's passed (cause I need to control when the volume is reinitialized and when it isn't). Dependency onrsync
prevents me from usingFROM: scratch
filesystem-only image (which would be ideal for my scenario)