Closed cunninghamd closed 6 months ago
I observed the same problem with the stop
command
Any updates on this? This issue seems exist longer than a year now.
I have the same problem with compose 1.29.2 on Linux (with down and stop) Is there any workaround for this?
I've started to enjoy the beauty of profiles when I hit this wall :-(
Just a note on Compose V2:
docker compose --profile test down
will only take down services selected by the configured profile, but will fail removing network. We probably should make the down process more flexible and keep network if other containers ("orphaned") for same project are detected.
Just a note on Compose V2:
docker compose --profile test down
will only take down services selected by the configured profile, but will fail removing network. We probably should make the down process more flexible and keep network if other containers ("orphaned") for same project are detected.
In my case, the container included in the profile went down and the network and other containers not included in the profile too. As if I weren't using profiles at all. I'm not using dependencies between containers that may explain this behaviour.
@ndeloof what you described is indeed the intended behaviour, but does not work with V2, so I don't think the V1 is relevant. --profile is ignored and everything is stopped
@silently just tested with V2:
$ docker compose --profile test --profile toto up -d
[+] Running 5/5
⠿ Container truc-toto-1 Running 0.0s
⠿ Container truc-toto-2 Running 0.0s
⠿ Container truc-test-1 Started 0.8s
⠿ Container truc-test-2 Started 0.8s
⠿ Container truc-test-3 Started 0.8s
$ docker compose --profile test down
[+] Running 3/3
⠿ Container truc-test-2 Removed 0.2s
⠿ Container truc-test-1 Removed 0.3s
⠿ Container truc-test-3 Removed 0.3s
⠿ Network truc_default Error
sorry @ndeloof I got it wrong, just tested with v2.0.0-rc.3 and I have the same behavior
just tested 2.2.3 and profiles still do not work when using docker-compose down ! All services in the compose file stop..
I use the latest raspberry pi os, with bullseye and docker, compose 2.2.3
My down command stops every service ! COMPOSE_PROFILES=$COMPOSE_PROFILES docker-compose -p $mypc -f $composefile --env-file $envfile down --remove-orphans
My up command works with compose_profiles COMPOSE_PROFILES=$COMPOSE_PROFILES docker-compose -p $mypc -f $composefile --env-file $envfile up -d --remove-orphans
Latest version 2..3.3 still got this problem...
Same here with v2.4.1
It even seems to have gotten worse, as I was previously testing with v2.0.1
and that one would only stop the containers with the selected profile + containers without a profile. This version just seems to stop everything instead, completely ignoring the --profile
setting.
I would also add that there should be a --remove-orphans
or similar feature, that would work with --profiles
. There was an issue about this that has since been closed due to inactivity.
https://github.com/docker/compose/issues/8432
Both these features are quite important and the fact that they are missing almost renders the --profile feature useless, or at least certainly much less useful than it can (and should) be.
The original motivation for profiles was to be able to filter the services that should be started by default, see the discussion in https://github.com/compose-spec/compose-spec/issues/97.
For this reason the initial implementation did only affect docker-compose
commands affecting the creation and starting of services, see https://github.com/docker/compose/pull/7930.
So docker-compose down
not considering profiles was an intentional decision for reasons described in https://github.com/docker/compose/pull/7930#issue-743282655, i.e. using profiles with the other subcommands would most probably cause behavior unexpected to the user.
E.g. should docker-compose down
The current behavior is 2., since otherwise docker-compose down
would ignore and not stop any running services that are assigned a profile while the user would expect that all services are cleaned up by this command. Also networks and volumes used by such services could then not be removed since there would still be running containers using them.
I'd like this command:
docker-compose --profile web down
to only stop/remove services associated with the profile specified, i.e.: web.
For this to work the behavior 1. would need to be implemented. But according to the spec
Services without a
profiles
attribute set MUST always be enabled.
I.e. the above command would still not only stop/remove services associated with the profile specified, i.e.: web, but all services with the associated profile and all default services - which again is most probably unexpected behavior for the user.
Observed result
When
down
is run, ALL services are stopped/removed.
So currently this is actually the expected behavior and in compliance with the spec.
I would also add that there should be a
--remove-orphans
or similar feature, that would work with--profiles
. There was an issue about this that has since been closed due to inactivity.
Since all defined services are part of the Compose application model and --profiles
only affects which of them are enabled, services that are disabled are not considered orphans because they actually do exist in the application model.
Actually, using compose v2, you can easily address both use-cases:
[x] stop and remove only services that are enabled - i.e. have no explicit profile assigned
docker compose --profile xxx stop
will load compose project and activate profiles, then only stop selected services
[x] stop and remove all services regardless of the profile
`docker compose --project-name
In order to change the current behavior to achieve the desired results of this issue probably differing behaviors would be required to match with user expectation, which should be
docker-compose down # stop and remove *all* defined services - as before
docker-compose --profile web down # stop and remove only services with the `web` profile
But the latter introduces new problems with services that are assigned to multiple profiles: should docker-compose --profile web down
stop the services which are assigned only to the web
profile or all services that are assigned to the web
and any other profiles? What about the services enabled by default?
E.g. with this compose.yml
services:
frontend: # our main application
# no explicit profiles, i.e. default
database: # a test database for local development
profiles: ["dev", "debug"]
db_frontend: # a frontend to inspect the test database
profiles: ["debug"]
depends_on:
- database
docker-compose up # start only main services, i.e. frontend
docker-compose --profile dev up # start frontend and database
docker-compose --profile debug up # start frontend, database and db_frontend
Now, what should docker-compose --profile debug down
do:
frontend
, database
and db_frontend
- which would be according to the specdatabase
and db_frontend
- i.e. only services having the debug
profile, which is not in compliance with the specdb_frontend
- i.e. services having only the debug
profile assigned?While 1. would not solve this issue here, 2. would come unexpected to the user with
docker-compose --profile dev up # start frontend and database for development
docker-compose --profile debug up # additionally db_frontend
docker-compose --profile debug down # intent to only stop db_frontend, but also stops database
whereas 3. would be unexpected with
docker-compose --profile debug up # start database and db_frontend
docker-compose --profile debug down # intent to stop all services started with previous command, but database is still running
Also, what about docker-compose --profile dev down
: respecting profiles here would exclude db_frontend
and only stop database
(and frontend
), but since db_frontend
depends_on
database
both should be stopped
First of all thank you for your rather quick replies.
Replying to @acran
I would also add that there should be a --remove-orphans or similar feature, that would work with --profiles. There was an issue about this that has since been closed due to inactivity.
Since all defined services are part of the Compose application model and --profiles only affects which of them are enabled, services that are disabled are not considered orphans because they actually do exist in the application model.
I understand that by definition they aren't orphaned as they still are described, but the argument still stands.
Replying to @ndeloof
Actually, using compose v2, you can easily address both use-cases:
[x] stop and remove only services that are enabled - i.e. have no explicit profile assigned
docker compose --profile xxx stop
will load compose project and activate profiles, then only stop selected services[x] stop and remove all services regardless of the profile `docker compose --project-name stop`` will look for actual project resource and stop them. Original compose file is not loaded
I would argue that the most important and obvious use case is missing. We have base services and services that were started with more than 1 enabled profiles. Now one of the profiles has actually been disabled and there's no easy way to remove the services that should no longer be running.
I struggle to see a case where we are interested in removing the base services + the services with profiles we have enabled but NOT the services with profiles that have since been disabled.
Replying again to @acran because in the meantime there is a new comment:
But the latter introduces new problems with services that are assigned to multiple profiles: should
docker-compose --profile web down
stop the services which are assigned only to theweb
profile or all services that are assigned to theweb
and any other profiles? What about the services enabled by default?
I agree with this which is why I feel that the most useful feature that is missing a support for a --remove-orphans
equivalent. I.e. remove what is no longer relevant/enabled.
I agree with this which is why I feel that the most useful feature that is missing a support for a
--remove-orphans
equivalent. I.e. remove what is no longer relevant/enabled.
This then should be a separate flag, something like --remove-inactive-profiles
or --remove-disabled
, to keep a clear distinction between orphans (missing in the configuration) and disabled services (disabled by profiles).
I agree with this which is why I feel that the most useful feature that is missing a support for a
--remove-orphans
equivalent. I.e. remove what is no longer relevant/enabled.This then should be a separate flag, something like
--remove-inactive-profiles
or--remove-disabled
, to keep a clear distinction between orphans (missing in the configuration) and disabled services (disabled by profiles).
I do understand that technically they are not orphans and so it's not completely unsurprising that --remove-orphans
doesn't work. The main issue is that there's not alternative. I do agree with you that having something --remove-inactive-profiles
or --remove-disabled
is the best approach, hence the equivalent in my reply. This would work wonderfully in our use case and in my opinion would be a great addition overall to the profiles
feature.
In fact I see use cases in which both of them can be used so it makes perfect sense that it would be a separate (not exclusive) flag.
Is this something that we can add to this project?
Like I mentioned on slack feel free to let me know if I can be of help, but unfortunately I have no experience with go which is why a PR hasn't been made by me :sweat_smile:
BTW, apologies if I'm polluting this conversation too much but it would also be very useful (and IMO it should also be a thing) if profiles were added as labels so that they would show up in docker inspect
, and also so that they could be used with --filter
commands for example. Just a suggestion :D
@acran @ndeloof Any updates on this? I was wondering if there has been any progress on this front since our last discussion.
Please let me know if I should create a separate issue or if I can be of any help.
Thanks!
I plan to experiment extending --remove-orphans
so it includes services created by profile, that is disabled when later running compose up
I plan to experiment extending
--remove-orphans
so it includes services created by profile, that is disabled when later runningcompose up
From what I understood @acran was arguing for not extending --remove-orphans
as technically services with profiles that aren't enabled, technically aren't orphans. I can't say I disagree with this. His approach was to have a separate flag like he mentioned above.
Either way as long as there's progress I'm happy. Please tag me if/when you implement this feature. And as before if I can help in any part do let me know. I'm not sure how hard of an implementation this would be, but if it's not super hard, I could maybe do the PR provided I get some pointers to get me started.
Was also deceived by this earlier today when running docker compose up -d --remove-orphans
(w/ env var COMPOSE_PROFILES=test
) which did not remove the inactive profiles/containers as I thought it would. So extending the orphans command or adding a new command would be much appreciated. With the way we are planning on using these profiles, if we adjust the profile list in the env file, we will have containers persists that should be destroyed after re-running the docker compose up
, therefore requiring a second command that specifically targets the orphaned/disabled container, assuming we remember to do so. Seems like there is enough information to understand the user's intent to remove the container tied to that one profile, however, I realize this gets more sophisticated with dependent services and such.
This problem still exists? any updates on it?
yes...
when i use profiles to start services from a compose file the right services start.. But when i want to close a part of the services that are running with profiles, everything stops....
I see that unfortunately this hasn't been moving.
@laurazard you self-assigned this. Is there something I or someone else can do to move this along?
@ndeloof did you end up making any of the experiments that you mentioned?
As I mentioned before, I would be happy to help but at least some guidance or a place to start would be appreciated.
Thank!!
Ah apologies, I meant to look into this when I did but couldn't find the time to do so. Maybe @ndeloof @glours or @milas can TAL at it.
Some of the comments above led me to believe that docker compose --profile foo down
only removes foo
. But a basic test in version 24.0.5
proves it removes all services. So it doesn't work in compose v2 either.
If you land here: don't spend time tinkering with the various workarounds... just change your design.
Follow-up to my above comment about avoiding profiles and "just change your design".
I did that: instead of profiles, I used replicas: 0
and scaled services on demand. But that too has an issue, which is that when scaling a service up/down, it stops all the other services.
I wonder whether it's better to use profiles (but have difficulty with docker compose down
), or scaling (but have difficulty with side effects). Either approach would be fine, but both approaches have issues.
If anyone comes up with a clever workaround (until either approach is feature-complete), please post your solution here?
I was going to start using such compose « profile » feature, but unfortunately after reading current thread, it seems to be bringing more issues on the table compared to what it really brings.
Or it’s just an incomplete/partially developed feature (still a wip??) where the main focus has only been around the « up », leaving all the other existing compose commands without good support for profiles, making it irrelevant for most scenarios.
@lonix1, I agree with you and I’ll be heading toward avoiding using compose profiles entirely, and change the approach to this common problem of grouping a subset of compose services together, and then ability to « up/start/stop/down/etc » them.
I was thinking of either one of these legacy approaches:
1- Classic split of main compose yaml file into multiple ones, so that instead of passing profile(s) to your compose commands, you pass compose yaml file(s).
2- Keep your main compose yaml file as is (no split like in 1-), and manage your groups of services separatly/manually (outside compose landscape basically..). There are multiple ways to achieve this, a few examples could be either having a group/list of compose service names in a separate txt file, or that list of services directly in a shell script variable, it depends on your use case. Then you can provide any of those lists of services to the compose commands:
// services are space-separated
docker compose start [OPTIONS] [SERVICES…]
docker compose down [OPTIONS] [SERVICES…]
Of course, each approach has its pros & cons, but perhaps those alternate solutions have a better overall score than the pros & cons of the current partial compose profile solution. Again, it depends on our/your use cases..
I seem to have an extra problem with this, by docker-compose contains a custom network, and multiple profiles.
If I "up" 2 profiles, one at a time, and then "down" one of them, the network gets removed, and I get this error trying to up anything:
Error response from daemon: network XXXXXXXX not found
The original issue has been addressed and discussions here start going in random directions, which make it impossible for us to offer triage and priorization on the possible fixes we can provide. So I'm closing this issue, you're welcome to create a new one with an explicit use-case as a bug report or feature request
@RangelReale please open a dedicated issue then
@Frank-D "leaving all the other existing compose commands without good support for profiles" - please clarify what you would expect other commands to offer in a dedicated "feature request" issue
Description of the issue
I have a set of services that I'd like to generally be "always on", with a subset that I'd like to restart as needed.
Each of these commands works as expected:
docker-compose --profile web up -d
// starts services associated with "web" profile -- works finedocker-compose --profile infrastructure up -d
// starts services associated with "infrastructure" profile -- works fineI'd like this command:
docker-compose --profile web down
to only stop/remove services associated with the profile specified, i.e.: web.Context information (for bug reports)
Output of
docker-compose version
Output of
docker version
Output of
docker-compose config
(Make sure to add the relevant-f
and other flags)Steps to reproduce the issue
-profiles
todocker-compose.yml
docker-compose --profile XXX up -d
docker-compose --profile XXX down
Observed result
When
down
is run, ALL services are stopped/removed.Expected result
When
down
is run, only services matching--profile
are stopped/removed.Stacktrace / full error message
Additional information
docker-compose
installed viacurl
(since 1.28.x isn't part of Docker Desktop for Mac yet).Love the possibility that profiles preset! Thanks for your hard work!