Proposal: make project-name persistent.

[thaJeztah]

By default, Compose bases the project name on basename of the directory compose commands are run from. The project name can be overridden either by passing a -p / --project-name option for each command or setting the COMPOSE_PROJECT_NAME environment variable.

Using the --project-name option for each command is error-prone and for- getting to pass the option when doing (for example) docker-compose up, will result in another instance of the project being created under a different name or even containers of a different project being replaced.

Using the COMPOSE_PROJECT_NAME environment variable is not useful when dealing with multiple projects and can cause similar problems.

Proposal: make project-name persistent

To solve these problems, compose will save the name of a project to a file in the build-context when the project is first started / built.

If a project-file is found, compose automatically uses the project-name from that file and throw an exception if the user is trying to override the project-name using the --project-name option.

File location

To allow further expansion of options, compose creates a hidden .docker-compose directory in the "root" directory of the build-context. Inside that directory, a project-name file is created, containing just the name of the project;

tree -a
.
├── .docker-compose
│   └── project-name
└── docker-compose.yml

Request for comment

There are a couple of things left to be discussed;

• Should the configuration-file be created automatically or should this be an explicit action by the user (e.g. docker-compose init --project-name=foobar)
• Should a command be added to remove the configuration file(s)? (e.g. docker-compose destroy)
• Compose allows specifying an alternative compose-file (--file) should the project-name also be applied to those? Should each compose-file get its own configuration?

And, in a wider scope;

• Is the project-name actually important? If not, compose could create a random project-name and store that inside the configuration. This would greatly reduce the risk of conflicting names with other projects running on the same server.

edit: renamed Fig to Compose

[timgriffiths]

+1 be keen to see this feature

[deizel]

@timgriffiths, looks like this is possible in the latest RC by adding an environment file to your project:

• https://docs.docker.com/compose/env-file/
• https://github.com/docker/compose/releases/tag/1.7.0-rc1

[timgriffiths]

@deizel So i looked at the environment file, but i still think there is a good use case for being able to define project name within the compose file.

In our situation is we have Prod, Staging, UAT, Dev compose files just so we can start up different version of our application stack, and sometimes we want to stand up a Staging and UAT environment on the same swarm, so i can use environment variables or what not but that relies on the user all remembering to set this, where as if it was in the compose file everything would be checked into the repo and we could keep everything inline and simple to explain across all the users.

[edevenport]

+1 We have a situation where our compose file has references to a preconfigured remote volumes. Compose automatically adds the project name to those volume names which then fail to mount due to mismatched names. Being able to explicitly set the project name in the compose yml file would make the naming convention much more obvious to others involved in the project rather than hiding environment variables in obscure external files that might vary across environments.

[fesor]

@edevenport in your case you could use external option of named volumes:

# docker-compose.prod.yml
volumes
  dbdata:
    external:
      name: my-project-db-data

[edevenport]

Thank you @fesor - I should have gone into more detail. I am actually mounting glusterfs volumes using a configuration similar to this:

    ...
    volumes:
      - media:/data/media:ro

volumes:
  media:
    driver: glusterfs

In this case, media becomes projectname_media on the host and fails to mount unless the glusterfs volume is created ahead of time with the project name. I had thought mounting the glusterfs volumes on the host manually and using external in docker-compose file, but I would rather not have to do that manually across all nodes ahead of time.

[fesor]

@edevenport I understand that. But you could just add this volume manually with docker volume create and use it:

    volumes:
      - media:/data/media:ro

volumes:
  media:
    external:
      name: my-glusterfs-media

So there is no need for changing project, and get rid of any prefixes.

@timgriffiths's case is something more complex, and it doesn't have any workarounds. I use simple wrapper around docker-compose just to not remembering project names.

[timgriffiths]

@fesor so we have worked around the problem by just sticking the compose files in descriptive folder names, this gets around this limitation. But would be very nice to see it added in later releases

[fesor]

@timgriffiths I made a PR (https://github.com/docker/compose/pull/3118) - maybe this would help. But it doesn't handle your case if you have multiple compose files.

[timgriffiths]

@fesor that PR would be perfect is it a possibility to get it merged?

[mkuzmin]

Now Compose provides Environment file, so the project name can be persisted there.

[fesor]

@mkuzmin it's sad that there is COMPOSE_FILE but there is no COMPOSE_OVERRIDE_FILE.

[schmunk42]

@fesor AFAIK you can specify multiple files

[fesor]

@schmunk42 I could specify project name either. But this doesn't solve problem. I want to have in deploy script something like this:

docker-compose up -d

instead of

docker-compose -f $COMPOSE_FILE -f $COMPOSE_OVERRIDE_FILE \
            up -d

COMPOSE_PROJECT_NAME in example above is specified by CI. And feature like .env file is cool but... useless in case of project name persistence.

I use .env to DRY env variables (and for docker-compose < 1.7 I have simple wrapper around), but it doesn't solve any other issues. @timgriffiths already pointed on case with deployment on swarm cluster.

[dnephin]

COMPOSE_FILE=one.yml:two.yml is how to set multiple files. So just include the override file in $COMPOSE_FILE

[fesor]

@dnephin missed that feature, cool.

Well... about swarm deployment, I already handling that via COMPOSE_PROJECT_NAME in my jenkin's job ENV variables so problems only with manual deployment. And my implementation doesn't allows default project name to be overridden so...

[mikehaertl]

This issue is almost 2 years old and still unresolved. I wonder what exactly is hindering the docker team to finally add a proper fix. Is it really so hard to add a simple new configuration directive for docker-compose.yml files?

While the workaround with .env is nice it's not alway usable (the .env file may already be used by your application). And it's also still a workaround.

And I've even seen other new issues which may be related to the project name creep into latest versions (#3966).

So what is the problem really?

[aanand]

Is it really so hard to add a simple new configuration directive for docker-compose.yml files?

No! The issue has never been difficulty of implementation.

While the workaround with .env is nice it's not alway usable (the .env file may already be used by your application). And it's also still a workaround.

It's not a workaround, it's a solution! The .env file contains environment variables that are to be read by Compose. If you have environment variables that are to be read by your application, put them in a different file (say app.env) and reference it using the env_file option.

So what is the problem really?

Adding the project name to a docker-compose.yml file makes it less portable, which is not something we want to encourage. Now that there's a way (with .env) to persistently specify the project name without sacrificing the portability of docker-compose.yml, the case for adding it in as a configuration option is weaker. That's the primary reason this issue hasn't moved.

[mikehaertl]

Adding the project name to a docker-compose.yml file makes it less portable,

Couldn't this be optional? Personally I never check in my docker-compose.yml but only supply a docker-compose-example.yml in my repo which I tweak locally. E.g. during development I may add some extra env vars or map additional volumes into my container. Why not also give us the option to specify the project name?

Following your argumentation, you should also move all the network options to .env as they usually are not portable at all and depend on the network setup of your host machine.

Put another way: What makes the project name so special? There already are many other options in the docker-compose.yml that are not really portable. Why not give the developer the option to choose whatever is best for his use case?

[axsuul]

I also think the option should be made available. As Docker use becomes more wide-spread, projects with only one developer using containerization within their workflow will become more and more common. In those use cases, the project name being constant is fine. A good example of another project allowing flexibility like this is vagrant with their Vagrantfile. They realize that there are certainly use cases for large teams but then also recognize the lone wolf developer scenarios.

I think implementing a project name option here is important for adoption. It could throw off solo developers that don't care about a "scalable project name configuration strategy".

[simonvanderveldt]

Adding the project name to a docker-compose.yml file makes it less portable, which is not something we want to encourage. Now that there's a way (with .env) to persistently specify the project name without sacrificing the portability of docker-compose.yml, the case for adding it in as a configuration option is weaker. That's the primary reason this issue hasn't moved.

How does adding the project name to the docker-compose.yml make it less portable?

IMHO it's actually the contrary, as shown by #3966 created by @mikehaertl. This issue clearly shows that not having the project name stored in docker-compose.yml makes things actually less portable (as in: having a bigger probability to clash with other compose projects started from the same machine) since Compose relies on a convention about the container folder name that many people, at least initially, won't know.

Adding adding an .env file not only adds an additional thing to learn for new people that want to adopt Compose but since I'd also have to add it to my repo next to docker-compose.yml to prevent these collisions I don't see how that would be any different portability wise than just defining the project name in docker-compose.yml.

[fiveanddone]

I'm also for adding the name into docker-compose.yml. The .env file is used in other frameworks I use and it's important that remains off the repo. However, for maintaining localized containers across my developers, the less they know about docker ( at first ) the better. I want a simple entry point that won't collide with other projects. I don't want to explain why things are the way they are. Just a simple

clone repo docker-compose up

My workaround now is still to create the file that @schmunk42 proposed.

[joequery]

The container itself shouldn't care about the project name. Why use environment variables as a mechanism only intended for the host to care about?

[thasmo]

I'd love to see support for this in the docker-compose.yml file. If one needs to set it via environment variable, it can be defined within the .env file (if not set directly on the host) and used via variable substitution in the docker-compose.yml file.

[hsimilli]

If named containers are allowed why would named projects not be allowed?

The use case where a project name definition in the .yml file would be handy is the following: We have the same docker-compose file for different webservers (different database credentials, different data-volumes, some small differences like logo, etc). There are 3 services per application running, and they are configured in a compose file. When multiple servers are run next to each other, it would be nice to see which service belongs to which project.

Variable substitution in the container name comes in handy, with the $COMPOSE_PROJECT_NAME as prefix. Well I understand the .env-file solution, but this makes it not more "developer-friendly" in a (CI) deployment environment.

[RobIsHere]

Having one docker-compose kill containers of a totally different docker-compose because there was no -p or environment variable set and you created the docker-compose in a "docker" subfolder is a critical bug in my opinion.

There is simply too much room for error in serious deployments. Just forget to specify the project name, whether in -p, .env or elsewhere: you go offline for some time until you notice what happened. Even worse: the colliding service goes offline, not the one you are working on. You will have a happy day :(

[mikehaertl]

@dnephin What exactly is holding up the docker team to finally implement this? How many more complaining users are required? This is such a trivial fix that it's hard to believe, that still nothing has happened.

[schmunk42]

Maybe I am the only one here who has some concerns with this, but anyway. In our setup, we modify only .env and app.env files for switching environments, but this relies on having multiple files and merging yml files.

If this gets implemented, please think about BC.

Eg. if a project name is defined in docker-compose.yml and .env the latter should take precedence. Otherwise people who rely on managing this via .env, get simliar problems with overlapping stacks, like with the current workflow and relying on folder names.

[mikehaertl]

@schmunk42 We're not saying, that the .env feature should be removed. If you prefer to have your project name in .env then simply don't configure it in docker-compose.yml.

But others prefer to have it in their docker-compose.yml as the discussion here clearly shows. They should be able to do so, if they want. It's an optional feature. What takes precedence if both are configured is a matter of defining a simple convention.

[davidjeddy]

Srsly, please add named projects. Use case: Project one: docker-compose up --build --remove-orphans

• nginx
• static content Project two: docker-compose up --build --remove-orphans
• nginx
• static content

When project two starts it kills project one's containers with a exit 137 (code 128 + level 9).

Now I have to run docker system prune and rebuild networks every day to keep myself from having dozens of dead containers.

[dnephin]

export COMPOSE_PROJECT_NAME=somethingnew

[mikehaertl]

Can someone from the core team maybe finally explain, what is holding up this feature? It's so frustrating, how things that are so easy to fix are blocked for no good reason.

The only argument so far was to keep the docker-compose.yml portable. This makes no sense, because

• project name in the composer file does not automatically make it not portable anymore
• not everyone even has this requirement
  • we already have other options that can limit portability in the same way: networks, port, ...

It all very much depends on the specific use case. But just because some don't want this option shouldn't mean that everyone has to follow their opinion!

[fesor]

@mikehaertl You already have persistent project names. Just put .env file and set COMPOSE_PROJECT_NAME variable there. I don't see any benefit in project name in compose file anymore.

[thasmo]

@fesor I generally dont' have .env files under version-control because they contain machine/environment-specific settings - that's also why I'd like to see project names being configruable in the docker-compose.yml file.

[nhooey]

@fesor: I think @thasmo is correct on this. The project name should have the option of being specified in the compose.yml file because if it's relevant to all developers using the project, it should be in version control.

[mikehaertl]

@fesor I'm sorry, but I don't think that personal preference is a real argument. The real question is not, why we don't want to use .env or environment variable. It's the other way round: Why was it not put in the config file where it belongs?

Almost all options that you can pass to docker on the command line can be specified in that file. Only the project name is for some mysterious reasons excluded. Is this a form of apartheid among arguments? :P I claim equal rights for all of them! Let the developer decide and don't put artificial constraints on them.

And again: No one is taking away the old option from you - we only finally want this extra option. That's only fair.

[nhooey]

@dnephin: It seems as though you and everyone else is happy with the solution stated in cr7pt0gr4ph7's refinement of this issue.

Would you be willing to accept a pull request that implements it, or do you intend to have one of the maintainers write it?

@cr7pt0gr4ph7

[davidjeddy]

@dnephin @fesor having the project name in the *compose.yml facilitates a unified configuration schema

[dnephin]

It's not clear to me why the project name matters. No part of the config file should rely on any specific project name.

The only important quality of the project name is that it doesn't conflict with the name of any other project, which is actually an argument against putting it in the docker-compose.yml file. A developer on a project will often use a different directory name for each project, so the default is often correct for most use cases.

When the default isn't correct there is a way for the developer to override it (-p, or COMPOSE_PROJECT_NAME), which can either be aliases or put into the .env file.

I'm not against a project file that would define the project name, I'd just like to understand why this feature is so critical for some. If it's because the Compose file requires a specific project name, I see that as a separate problem that should be addressed differently,

[RobIsHere]

I wanted to have 4 compose files in 1 directory. The directory name is wrong by default here. The only option I have is .env that only works for 1 compose file in 1 directory.

_Why I rule out COMPOSE_PROJECTNAME and -p: The COMPOSE_PROJECT_NAME and the -p are in my opinion not save in production as you have to remember to set them. Or create a startup script and remember to not use docker compose up directly. When Person A relies on this mechanism and Person B doesn't know it, that is a certain failure. (I already detailed it above)

[fiveanddone]

@dnephin For me and my team, application logic is usually stored in an htdocs folder in our project folder. This is allows us to keep other related documents/resources together in one directory. As a dev ops working with remote developers. It's easy for me to have them adopt Docker if I don't have to assume anything about their folder structure. I've stated this in the past, .env is not an option for me because it's typically not part of the repo we all share.

[mikehaertl]

It's not clear to me why the project name matters. No part of the config file should rely on any specific project name.

It happened more than once to me, that I did a 'docker-compose down' on a host - and had a different container go down than the one I wanted! Just because I didn't remember, that there was also configuration in a .env file.

The only important quality of the project name is that it doesn't conflict with the name of any other project, which is actually an argument against putting it in the docker-compose.yml file. A developer on a project will often use a different directory name for each project, so the default is often correct for most use cases.

Maybe that's true for you - it is not for me and for many others. My app structure is like myapp/app/docker-compose.yml. So all my apps share the directory name app. And I have more than 1 container on a host.

When the default isn't correct there is a way for the developer to override it (-p, or COMPOSE_PROJECT_NAME), which can either be aliases or put into the .env file.

Seriously, I start to feel like I'm in a Kafka novel here. Isn't it obvious, that you can put basically all the command line options for docker into a docker-compose.yml - except for this one big exception?

Instead of asking us over and over and ignoring our answers, why can't someone finally justify the resistance. And no: " ... but I don't need it" is not a valid argument!

[kachkaev]

@dnephin I have multiple projects in their own git repos and in order to keep the root project directories clean, I put all docker-related stuff to a subfolder called docker (build context just becomes .., all works beautifully).

To run multiple projects on a single server, I currently have to create docker/.env.dist in each repo and cp docker/.env.dist docker/.env after git clone. Otherwise, project name is just docker, which I obviously don't want. In a few cases .env contains passwords etc, so copying .env.dist is required anyway, but in most of the cases .env exists simply because there is no way to define COMPOSE_PROJECT_NAME in docker-compose.yml.

I understand that there may be glitches in how the containers are started and stopped if the same COMPOSE_PROJECT_NAME is used twice inside docker-compose.yml, but that's already happening if I forget to copy .env.dist or when I accidentally assign the same name in two different .env files on the same server. For me it would be ideal if I could define default_compose_project_name right in docker-compose.yml and then override the value in .env if, let's say I want to run a staging copy of some project. This would be 100% BC, but more convenient.

[schmunk42]

I've stated this in the past, .env is not an option for me because it's typically not part of the repo we all share.

Maybe, that's the root-cause of it all, I said before that docker-compose "hijacked" this file and we were forced to rename our stuff to app.env.

Maybe docker-compose.env would have been the right choice.

FWIW: We're only changing .env files in production, the yml files are merged with a docker-compose.override.yml if needed.

A workaround is also to define your yml files in a way, that they do not start without a .env file, eg. using a variable image: $IMAGE.

[RobIsHere]

This is a workaround for my 4 docker- compose files in 1 directory case above @schmunk42 ? Really?

[schmunk42]

@RobIsHere But you need to use -f anyway, right?

[nhooey]

In reply to @dnephin in this comment:

It's not clear to me why the project name matters.

It matters because it affects the name of the Docker containers that docker-compose manages. If you forget to set the project name, docker-compose ps won't find the containers that you've created before with docker-compose --project-name <project_name> up -d <container_name>.

Also, when you run the global command docker ps, the project name will be included in the list of containers that are running, so you know where they came from. For example, if you are testing several code projects at once that all use MySQL, and they each have a mysql container, then docker ps will show an ambiguous list of containers. So, the project name is important in the context of many Docker projects running on the same machine.

No part of the config file should rely on any specific project name.

Having to set the project name in a different place than every other variable associated with the Docker Compose project doesn't make sense.

The only important quality of the project name is that it doesn't conflict with the name of any other project, which is actually an argument against putting it in the docker-compose.yml file.

For reasons stated in my first paragraph, this isn't the "only important quality of the project name". It's for ease-of-use and understanding.

A developer on a project will often use a different directory name for each project, so the default is often correct for most use cases.

In the non-default case where a developer puts all related Docker files in the docker directory, which is a perfectly reasonable way to organize a Docker project, the magic that sets the project name will create conflicting project names. So the same conflicts that you mention occur in this particular case. You don't need to protect the developer against project-name collisions when they are explicitly setting the project name anyway.

When the default isn't correct there is a way for the developer to override it (-p, or COMPOSE_PROJECT_NAME), which can either be aliases or put into the .env file.

Setting environment variables is an extra level of complexity that isn't necessary. When you're sharing a project between developers with version control, it's makes sense to use only files. Having to include command-line params with each invocation is tedious and error-prone. And the .env file can't be shared in version control because it's supposed to contain user-specific variables, not project-specific ones. Therefore, the current ways to set a project name are insufficient.

I'm not against a project file that would define the project name, I'd just like to understand why this feature is so critical for some. If it's because the Compose file requires a specific project name, I see that as a separate problem that should be addressed differently,

All variables that affect the functioning of docker-compose should go in the same place, the docker-compose.yml file. The current ways of setting a project name introduce unnecessary complexity.

Just about everyone subscribed to this issue agrees with these points.

Adding the ability to set the project name in docker-config.yml won't even conflict with any current functionality. There isn't a reason not to implement it.

[fiveanddone]

@schmunk42 Last time I checked, docker didn't allow us to pick the name of our .env file. Did I miss something ( it's hard to keep up with the changes ). Changing it to something like docker-compse.env would fix it for my use case. The problem I have is that I'm using two very popular technologies that both require .env file. PHP Laravel framework does not track that file in the repository and in production environments it sometimes doesn't exist.

I found this to be a real stubbling block when first adopting Docker because like @RobIsHere illustrates. All of my project directories are in themyapp/htdocs/docker-compose.yml format. In early days adopting Docker I accidentally deleted other containers that I didn't intend to. It would have ben better if docker named it's project randomly rather then using the folder name in my case.

I have several other remote developers that are starting to adopt Docker. As a team, it's always easier for me to instruct these developers to always/only docker-compose up to get these applications running. The less these early adopters know about Docker the better. Once they see the power behind it, they'll pick up on their own.

[dnephin]

So to summarize, the use cases seem to be when the default isn't appropriate. There are already ways to override the default, but those could be error prone and not obvious to early adopters.

Cases where the default isn't appropriate are:

• multiple compose files in the same directory (already requires you to specify -f to run them)
• using the same directory name for all projects (I guess this also requires you to specify -f, possible workaround is to use a unique directory name).

[fiveanddone]

@dnephin I would add, one less possible stumbling block for early adopters of Docker.