Add ability to select a docker-compose file

[CameronDiver]

Given a directory with more than one docker-compose file, users would like the ability to be able to select one or another of them.

Probably the best way to do this would be a flag to balena push.

---

Please keep in mind that we try to use the issue tracker of this repository for specific bug reports & CLI feature requests. General & troubleshooting questions are encouraged to be posted to the balena forums where the community can both contribute and benefit from the answers.

Before submitting this issue please check that this issue is not a duplicate. If there is another issue describing the same problem or feature please add your information to the existing issue's comments.

Front conversations

[pdcastro]

Connects-to:

• 587 Let users specify a Dockerfile with balena build -f

• 1704 Add support for using parent context in balena build

[MathiasKoch]

+1 on this feature

[balena-ci]

[pdcastro] This issue has attached support thread https://jel.ly.fish/#/support-thread~4115ae1d-c2da-41c6-bbe4-667e366c314f

[krasi-georgiev]

+1 on this feature

What are the chances of someone starting to work on this?

there is already a flag:--dockerfile so something like --composefile would be ideal

[robsonvn]

+1000 to this feature, I'm currently using bash script on my deploys just because of it.

[pdcastro]

@krasi-georgiev and others -- thanks for the feedback and upvoting. 👍 I am currently auditing and sorting all open issues in the CLI repo, and this issue is currently ranking in the top five. :-) Link: Sorted Backlog It may go down a bit as I find other high priority issues, but it will still end up in the top league. So, there is hope it will get implemented in the near future.

[balena-ci]

[camerondiver] This issue has attached support thread https://jel.ly.fish/#/b957d2b3-67b8-4b63-987a-2744af73a940

[rjhuijsman]

+1. We use different compose files to specify different environments, and having to use scripts is (1) a headache to develop, maintain, document (2) just lost me a bunch of time to even figure out.

[pdcastro]

Implementation note: Docker's documentation for the docker-compose.yml CONTEXT instruction states that the context path is "interpreted as relative to the location of the Compose file" (https://docs.docker.com/compose/compose-file/compose-file-v2/#context). Therefore, allowing the specification of an alternative docker-compose.yml file may also require changes to CLI dependency modules like resin-bundle-resolve, which may currently assume that the context path is relative to the "project root" (?). (It's probably also a good idea to test that the docker-compose tool actually behaves as documented.)

[hedss]

As an addition to this, as I can't see anywhere it's been made clear, the CLI should allow multiple -f references to different compose manifests. This then allows us to implement overriding functionality as per vanilla docker-compose. See here: https://www.flowdock.com/app/rulemotion/r-supervisor/threads/AIYlFQDc8BCdxT7MhyxosaatMaY

[pdcastro]

Update: I believe the main reasons why this issue hasn't been implemented yet are the complications and implications of the alternative docker-compose file being outside the "project root folder" - given that Docker's spec / docs state that the build context should be relative to the location of the docker-compose file. However, we could have an intermediate implementation step that allowed alternative docker-compose file names to be specified as long as they were located in the project root folder. This would make the implementation simpler, and I think we should do it. Additional discussion (private flowdock thread) cc: @srlowe

[erlend-aasland]

However, we could have an intermediate implementation step that allowed alternative docker-compose file names to be specified as long as they were located in the project root folder. This would make the implementation simpler, and I think we should do it. Additional discussion (private flowdock thread) cc: @srlowe

That would be a welcome feature, as it would suffice to solve our issues.

[jellyfish-bot]

[pdcastro] This issue has attached support thread https://jel.ly.fish/e1e54d6c-cac2-4eb8-b4fb-e07783107fa7

[cdgraff]

+1 on this feature

Really needed to avoid duplicate source code, or doing shell scripts that override docker-compose.yaml with the risk of this.

[edorgeville]

Re-commenting here after discussion on #2177 that shallow merges docker-compose.dev.yml:

Using docker-compose, you can specify multiple yml files using the -f flag. It uses a recursive merge as described here: https://docs.docker.com/compose/extends/#adding-and-overriding-configuration

For single-value options [...], the new value replaces the old value. [...] For the multi-value options [...], Compose concatenates both sets of values

I think this would be a preferable merge method, for parity. This way you can have a minimal docker-compose.dev.yml, for example, exposing a node debugger in local mode only:

services:
  app:
    ports:
      - "9229:9229"

[codethief]

Sigh. I just came across this issue because I created two different docker-compose files, docker-compose.dev.yml and docker-compose.prod.yml and thought I could just pass them to balena push via --dockerfile docker-compose.{dev|prod}.yml. Not surprisingly, I was astonished to read something about a "Docker compose dev overlay" in the log output of balena push.

I don't think https://github.com/balena-io/balena-cli/pull/2177 solves anything and, even worse it's implicit, unexpected behavior and nowhere documented. My development and production docker-compose files are completely different, so overriding stuff doesn't do the trick.

Update: I believe the main reasons why this issue hasn't been implemented yet are the complications and implications of the alternative docker-compose file being outside the "project root folder" - given that Docker's spec / docs state that the build context should be relative to the location of the docker-compose file.

I don't really understand the issue. balena push already comes with a --source option. Why can't this option automatically be set to the parent dir of the --docker-compose-file one provides?

[maggie44]

Sigh. I just came across this issue because I created two different docker-compose files, docker-compose.dev.yml and docker-compose.prod.yml and thought I could just pass them to balena push via --dockerfile docker-compose.{dev|prod}.yml. Not surprisingly, I was astonished to read something about a "Docker compose dev overlay" in the log output of balena push.

I don't think #2177 solves anything and, even worse it's implicit, unexpected behavior and nowhere documented. My development and production docker-compose files are completely different, so overriding stuff doesn't do the trick.

Update: I believe the main reasons why this issue hasn't been implemented yet are the complications and implications of the alternative docker-compose file being outside the "project root folder" - given that Docker's spec / docs state that the build context should be relative to the location of the docker-compose file.

I don't really understand the issue. balena push already comes with a --source option. Why can't this option automatically be set to the parent dir of the --docker-compose-file one provides?

Are you saying that when you do docker-compose up it automatically merges the docker-compose.dev.yml files? Or only when you specify it with --f file1.yml --f docker-compose.dev.yml?

[codethief]

On a somewhat more abstract level: @pdcastro Is there a (documented) way for me to talk to the balena supervisor on my local device directly? This is not the first issue I'm dealing with when it comes to Dockerfiles. Other issues I've encountered are:

• Dockerfiles can't be modularized. This is a long-standing proposal but it doesn't look like it's ever going to get implemented.
• Multi-stage builds together with the target option in docker-compose format v3.4 provide ways for modularization in a certain sense and in some cases but not really. Besides, the balena supervisor doesn't support v3.4 of the docker-compose format, let alone the target option.

All in all, these issues come down to the fact that we're forced to write entirely static docker-compose.yml files / Dockerfiles and that we're not able to programmatically generate them (short of echo "some Docker command" >> Dockerfile) and hand them over to the Docker engine through an API that is not the shell. I get that Dockerfiles and docker-compose.yml files must be of a declarative, not imperative (let alone Turing-complete) nature but why does this mean I can't imperatively generate a declarative file through, say, Python or TypeScript?

Basically, what I have in mind is writing a build script in Python that assembles my Dockerfile dynamically and hands it over to the balena supervisor on my target device / the build server together with the proper build context. (If it were the Docker engine, not the balena supervisor, the latter part could be achieved through e.g. https://github.com/docker/docker-py ) Dynamically composing my Docker containers would then be a piece of cake as docker-compose is only a wrapper around the docker CLI, anyway. Here's some pseudo-code of what I have in mind (if I wanted to talk to the Docker engine instead of the Balena supervisor, that is):

from this_package_doesnt_exist_yet import Dockerfile

d = Dockerfile()
d.FROM("balenalib/jetson-nano-ubuntu:bionic")
d.ENV(...)
d.RUN(...)
d.CMD(...)

if __name__ == "__main__":
    from docker import APIClient
    client = APIClient("my device IP")
    # Compare https://docker-py.readthedocs.io/en/stable/api.html#module-docker.api.image
    client.build(fileobj=d.as_fileobj(), custom_context=...)

[codethief]

Are you saying that when you do docker-compose up it automatically merges the docker-compose.dev.yml files? Or only when you specify it with --f file1.yml --f docker-compose.dev.yml?

You probably meant to write balena push instead of docker-compose up but, yes, it's the former:

$ ls -l | grep docker-compose
-rw------- 1 user user  284 Apr  1 18:10 docker-compose.dev.yml
-rw------- 1 user user  342 Apr  1 18:10 docker-compose.prod.yml
$ balena push balena.local
[Info]    Starting build on device 192.168.2.134
[Info]    Docker compose dev overlay detected (docker-compose.dev.yml) - merging.

I suppose you get my confusion. :)

[maggie44]

I did mean push.

Oh boy, yes, that's an issue for sure. Not only in it's step away from Docker etiquette - with Docker as our base documentation for understanding balena engine - but it is a breaking change for a bunch of repos when I next update as I have dev files named like that sitting alongside quite happily (until this).

Eek..

I see the premise of what is trying to be achieved here, but would suggest reverting to Docker formats post-haste: https://docs.docker.com/compose/extends/#adding-and-overriding-configuration

In short, it would mean automatically merging docker-compose.override.yml files, not docker-compose.dev.yml files. If someone wants to merge a dev file, they can simply name it docker-compose.override.yml. And we don't end up bricking everyone else. We can then apply our existing Docker understanding and docker docs to our use.

Might just be the one liner here, based on the search of the pull request @pdcastro : https://github.com/balena-io/balena-cli/blob/f3d750a0247ae0f272392666d9d243ed65e8e4e6/lib/utils/compose_ts.ts#L175

[codethief]

but it is a breaking change for a bunch of repos when I next update as I have dev files named like that sitting alongside quite happily (until this)

Oh wow, you're right. This is even worse than I had thought.

In short, it would mean automatically merging docker-compose.override.yml files, not docker-compose.dev.yml files.

I don't think a hard-coded, implicit file name is a good idea – it will only provide a solution in the simplest of use cases and even then it's not clear what it does if one doesn't read the Balena docs carefully. On top of that, it still deviates from docker's behavior. I would much prefer an explicit CLI option.

[maggie44]

but it is a breaking change for a bunch of repos when I next update as I have dev files named like that sitting alongside quite happily (until this)

Oh wow, you're right. This is even worse than I had thought.

In short, it would mean automatically merging docker-compose.override.yml files, not docker-compose.dev.yml files.

I don't think a hard-coded, implicit file name is a good idea – it will only provide a solution in the simplest of use cases and even then it's not clear what it does if one doesn't read the Balena docs carefully. On top of that, it still deviates from docker's behavior. I would much prefer an explicit CLI option.

An override by default file actually aligns with Docker practice, and is quite a handy feature. It's clear naming though prevents these issues happening accidentally: https://docs.docker.com/compose/extends/#understanding-multiple-compose-files

The additional feature that is also based on Docker, is to specify files with -f file1 -f file2 and them then merge, which is also useful although not really in the critical category for a feature.

But I think where this thread started, and would be most useful, is to just be able to specify the file you want to use '-f filename.yml' (after fixing the current merge issue). Also common Docker practice. Mostly because I can then have my development as docker-compose.yml and my production as docker-compose.prod.yml. I call my development .yml file far more than my prod, so people tend to have them this way around. I can then bring up whichever environment I need without having to rename files (which can lead to the issue of pushing those changes to my repo by mistake).

[pdcastro]

@srlowe, @joshbwlng, FYI, see 7 comments above (1st April 2021) regarding docker-compose.dev.yml, and PR #2241 Rename supported docker-compose overlay file

[maggie44]

Is there any update on this? It really is quite a significant issue, forcing non-standard naming conventions and bound to be tripping more people up down the line.

How about changing docker-compose.dev.yml to docker-compose.overlay.yml until the larger refactor planned is complete? It seems to align with what it is the feature does, and doesn’t clash with docker naming conventions.

[jamwest]

I'm running into the separate docker-compose file problem too. The compose specification has a profile option which if a flag was added for push/deploy/build could maybe solve some of these headaches?

[maggie44]

Any update on this? Being able to select a docker compose file (balena push -f docker-compose-non-default.yml) seems like such an important feature, standard in Docker, and is really making a mess of my development environments not having it. I am having to ask people to rename files from one to the other, and then inevitably someone accidentally pushes the wrong file back up to GitHub forgetting to rename them back again. There is already a --source for specifying a folder, why not a file?

All the thumbs up and length of this thread seems to suggest there is demand for it.

[danielmahon]

I am using an NX workspace mono-repo and having the ability to push a specific docker-compose would be great. Currently, I have multiple compose files "competing" in the root directory.

[jellyfish-bot]

[the-real-kenna] This issue has attached support thread https://jel.ly.fish/44dd2470-431d-4327-917b-51a7e22181ce

[hraftery]

In the meantime I had some success with a workaround:

1. move your project source files into a folder of their own, for example context.
2. make a copy of docker-compose.yml in that folder as well.
3. move Dockerfile.template into that folder and optionally make a copy for dev purposes.
4. add context: ./context to the build: section of your top-level docker-compose.yml file.

Now when you call balena push from the top level, it will behave as usual.

However, if you navigate to the context folder and call balena push, the docker-compose.yml from that folder will be used instead, where you can specify changes, including an alternate Dockerfile.

This worked for me by starting with balena-nodejs-hello-world, moving src and views into a new top level folder called context, copying the docker-compose.yml and Dockerfile.template files there too, and changing the top level docker-compose.yml file to:

version: "2"

services:
  balena-hello-world:
    build:
      context: ./context
    ports:
      - "80:80"

I was able to balena push <fleet-name> from the top-level as normal. I could then turn on local mode, cd into the context folder, and do balena push <ip-address>, and this time the project ran with live push enabled, except any changes I made in docker-compose.yml and Dockerfile.template files in that folder took effect.

One gotcha was that I also had to duplicate the package.json and package-lock.json files into the context folder, but maybe with a bit more care this could be worked around too.

[jellyfish-bot]

[hraftery] This has attached https://jel.ly.fish/469950c3-704b-4c50-8f2e-89db97b1f79f

[lmbarros]

@hraftery, concerning this comment:

One gotcha was that I also had to duplicate the package.json and package-lock.json files into the context folder, but maybe with a bit more care this could be worked around too.

I was able to use symlinks between the project root and ./context to avoid having physical copies of package.json and package-lock.json. Symlinking Dockerfile.template also worked well. This might a good option for anyone running a Unix-like OS on their development machines.

[Ahelsamahy]

I (along with ChatGPT ;) ) created a wrapper script for the Balena CLI, which addresses the issue.

The .py script moves the common folder from the current working directory (where the script is located) and then passes the next command to the Balena CLI as usual. One key feature is the comparison it performs before and after gracefully shutting down the script. It compares the copied common folder with the original one to detect any file differences (which is often the case when running in local mode). If a difference is found, the script will terminate, as it requires the user’s attention.

use python balena_wrapper.py pi_stuff push 192.168.1.100 --debug to run the script

BalenaCommonWrapper.zip