search

docker / compose

Define and run multi-container applications with Docker
https://docs.docker.com/compose/
Apache License 2.0
33.64k stars 5.19k forks source link

[BUG] env_file directive in include does not affect interpolation for service env_file #11741

Open ShadowLNC opened 5 months ago

ShadowLNC commented 5 months ago

Description

Within an include directive, I can specify an env_file. This changes interpolation for the included YAML, but if that YAML file has a service with an env_file, that environment file doesn't have its interpolation changed, and I believe it should.

Steps To Reproduce

docker-compose.yml ```yaml include: - path: include.compose.yml env_file: - .env - extras.env ```
include.compose.yml ```yaml services: mine: image: alpine command: env env_file: fakefile.env environment: COMPOSE_EXPLICIT: "abc" COMPOSE_FROM_MAIN: "${COMPOSE_FROM_MAIN}" COMPOSE_FROM_EXTRAS: "${COMPOSE_FROM_EXTRAS}" ```
.env ```env ENV_FROM_MAIN="env-frommain" COMPOSE_FROM_MAIN="compose-frommain" ```
extras.env ```env ENV_FROM_EXTRAS="env-fromextras" COMPOSE_FROM_EXTRAS="compose-fromextras" ```
fakefile.env ```env INTERPOLATE_FROM_MAIN="${ENV_FROM_MAIN}" INTERPOLATE_FROM_EXTRAS="${ENV_FROM_EXTRAS}" ```

Then, I run docker compose up to see the environment:

docker compose up (breaks extras.env) The `extras.env` file is missing from the env file interpolation step, but present on the YAML interpolation. ```text WARN[0000] The "ENV_FROM_EXTRAS" variable is not set. Defaulting to a blank string. [+] Running 0/0 ⠋ Container fake-mine-1 Recreated 0.0s Attaching to mine-1 mine-1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin mine-1 | HOSTNAME=4bbcdec26225 mine-1 | COMPOSE_FROM_MAIN=compose-frommain mine-1 | COMPOSE_FROM_EXTRAS=compose-fromextras mine-1 | COMPOSE_EXPLICIT=abc mine-1 | INTERPOLATE_FROM_MAIN=env-frommain mine-1 | INTERPOLATE_FROM_EXTRAS= mine-1 | HOME=/root mine-1 exited with code 0 ```
docker compose --env-file=extras.env up (breaks .env) The `.env` file is missing from the env file interpolation step, but present on the YAML interpolation. ```text WARN[0000] The "ENV_FROM_MAIN" variable is not set. Defaulting to a blank string. [+] Running 0/0 ⠋ Container fake-mine-1 Recreated 0.1s Attaching to mine-1 mine-1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin mine-1 | HOSTNAME=fd388d4b94b9 mine-1 | INTERPOLATE_FROM_EXTRAS=env-fromextras mine-1 | INTERPOLATE_FROM_MAIN= mine-1 | COMPOSE_FROM_EXTRAS=compose-fromextras mine-1 | COMPOSE_EXPLICIT=abc mine-1 | COMPOSE_FROM_MAIN=compose-frommain mine-1 | HOME=/root mine-1 exited with code 0 ```

In both cases I would expect both files (.env and extras.env) to be used because of the env_file directive under include, irrespective of the CLI --env-file argument. My expectation holds true for the YAML interpolation, but not when interpolating an environment file referenced by the YAML file.

Compose Version

$ docker compose version
Docker Compose version v2.26.1-desktop.1
$ docker-compose version
Docker Compose version v2.26.1-desktop.1

Docker Environment

docker info ```Text Client: Version: 26.0.0 Context: desktop-linux Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.13.1-desktop.1 Path: /Users/scott/.docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.26.1-desktop.1 Path: /Users/scott/.docker/cli-plugins/docker-compose debug: Get a shell into any image or container. (Docker Inc.) Version: 0.0.27 Path: /Users/scott/.docker/cli-plugins/docker-debug dev: Docker Dev Environments (Docker Inc.) Version: v0.1.2 Path: /Users/scott/.docker/cli-plugins/docker-dev extension: Manages Docker extensions (Docker Inc.) Version: v0.2.23 Path: /Users/scott/.docker/cli-plugins/docker-extension feedback: Provide feedback, right in your terminal! (Docker Inc.) Version: v1.0.4 Path: /Users/scott/.docker/cli-plugins/docker-feedback init: Creates Docker-related starter files for your project (Docker Inc.) Version: v1.1.0 Path: /Users/scott/.docker/cli-plugins/docker-init sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.) Version: 0.6.0 Path: /Users/scott/.docker/cli-plugins/docker-sbom scout: Docker Scout (Docker Inc.) Version: v1.6.3 Path: /Users/scott/.docker/cli-plugins/docker-scout WARNING: Plugin "/Users/scott/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/scott/.docker/cli-plugins/docker-scan: no such file or directory Server: Containers: 2 Running: 1 Paused: 0 Stopped: 1 Images: 41 Server Version: 26.0.0 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb runc version: v1.1.12-0-g51d5e94 init version: de40ad0 Security Options: seccomp Profile: unconfined cgroupns Kernel Version: 6.6.22-linuxkit Operating System: Docker Desktop OSType: linux Architecture: aarch64 CPUs: 8 Total Memory: 7.755GiB Name: docker-desktop ID: 85b60fc6-eedd-4b8c-996a-89f62a45ab85 Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 No Proxy: hubproxy.docker.internal Labels: com.docker.desktop.address=unix:///Users/scott/Library/Containers/com.docker.docker/Data/docker-cli.sock Experimental: false Insecure Registries: hubproxy.docker.internal:5555 127.0.0.0/8 Live Restore Enabled: false WARNING: daemon is not using the default seccomp profile ```

Anything else?

No response

ndeloof commented 5 months ago

Note: this require we re-implement WithServicesEnvironmentResolved within loadYamlModel so that env_file get loaded before we merge the included model with the including one. This is obviously feasible but need to handle with care as we already got a bunch of regressions in this area

clementgoclock commented 5 months ago

Got something similar here but not using include directive. Any interpolation variables from something else than .env is not working.

Exemple :

    env_file: vpn.env
    environment:
      HOMEPAGE_VAR_USER_NAME: "${USER_NAME:-Téléporteur pas configuré !}"
      HOMEPAGE_VAR_COHORT: "${SCHOOL_COHORT:-Sans Promo}"
      HOMEPAGE_VAR_VPN_NAME: "${VPN_NAME:-sans-vpn}"

Are using default values defined in environment. If i copy variables or rename vpn.env to .env it works.

Variables from vpn.env are correctly passed to the CT btw:

/app # printenv | grep COHORT
SCHOOL_COHORT=CacaHati
HOMEPAGE_VAR_COHORT=Sans Promo

@ShadowLNC could you try without include to check if interpolation is working with an env file different from .env ?

Compose version : 

Docker Compose version v2.26.1-desktop.1
ndeloof commented 5 months ago

@clementgoclock this is a distinct issue, please fill another one so we can track it. AFAICT this one would be hard to address, as you expect (higher-priority) environment to be resolved based on (lower-priority) env_file values. This would require some significant refactoring...

github-actions[bot] commented 6 hours ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.