`stack.env` does not work

[traverseda]

Bug description

Following along with the tutorial here: https://www.portainer.io/blog/using-env-files-in-stacks-with-portainer I get stack.env: no such file or directory

Expected behavior

stack.env should work as per the above blog post.

Portainer Logs

level=info msg="2022/03/28 12:47:58 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:47:57 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:52:49 http error: open /data/compose/57/stack.env: no such file or directory"
level=info msg=" (err=open /data/compose/57/stack.env: no such file or directory"
level=info msg=") (code=500)"
level=info msg=" (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:49:46 http error: open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:53:58 http error: open /data/compose/58/stack.env: no such file or directory"
level=info msg=" (err=open /data/compose/58/stack.env: no such file or directory"
level=info msg="2022/03/28 12:49:31 http error: open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:48:45 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:48:25 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"

Steps to reproduce the issue:

Be running in swarm mode.

Create a new stack with the content.

version: "3.8"

services:
  test:
    env_file:
      - stack.env

Add at least one environment variable to the environment variables section.

Try to deploy.

Technical details:

• Portainer version: Portainer CE 2.11.0
• Docker version (managed by Portainer): API version 1.41
• Platform (windows/linux): linux
• Command used to start Portainer (docker run -p 9443:9443 portainer/portainer): Uhh, it's been running for a while as a stack...
• Browser: Firefox
• Use Case (delete as appropriate): Using Portainer at Home
• Have you reviewed our technical documentation and knowledge base? Yes

Additional context

Assessing the possibility of running a dockerized mail server through portainer for my employer.

[zeh235]

same issue here

[derekpovah]

Seeing the same behavior on EE 2.12.1

[PrivateButts]

Ditto on EE 2.12.1, running on a swarm

When I go into the portainer vol, the compose folder that it's looking for doesn't exist. If I create the stack and then add env_file to the compose, it's never created in the compose folder created for the id.

If I create an empty file in that folder the stack is able to be updated with the env_file definition, but the file isn't updated with the env set in the gui.

[gaby]

Only way I have been able to get this to work is by passing

-v /opt/data/env.json:/opt/data/env.json to the portainer server container. That hack doesn't work with portainer-agent though, seems like the server tries to get the file from it's local fs instead of the agent fs.

[qosmio]

Same issue for 2.13.1 on EE. I had to revert to specifying environment variables manually... If anyone needs to quickly generate required YAML, you can use the following.

Switch to "Advanced Mode" copy everything into a file somewhere (i.e. /tmp/z).

EXAMPLE
$ cat /tmp/z
VAR1="TEST"
VAR2="TEST"

awk -F= 'BEGIN{print "    environment: "}{print "      "$1": \"${"$1"}\""}' /tmp/z

Which should output

    environment:
      VAR1: "${VAR1}"
      VAR2: "${VAR2}"

[JuniorC07]

to me, the error reason was the path of docker-compose.yml(using git repository), seted on Compose path field

Compose path: /some-project/docker-compose.yml

Before:

version: '3.4'
services:
  some-project:
    env_file:
      -  stack.env
    build:
      context: .
    image: some-project
    restart: always
    ports:
      - '8080:80'

After(Working): returning to the root directory with ../stack.env

version: '3.4'
services:
  some-project:
    env_file:
      - ../stack.env
    build:
      context: .
    image: some-project
    restart: always
    ports:
      - '8080:80'

Obs: I know it doesn't answer your question, but it could be related.

[gaby]

My problem is that when you specify env_file: myvars.env

The server works fine, but the agent uses the same env instead of its local one. :-S

[qosmio]

Obs: I know it doesn't answer your question, but it could be related.

Unfortunately, correct, I'm not specifying a local directory/project. Just letting Portainer manage everything (using its own volume) when I enter in the environment fields.

Another solution was to manually add the "stack.env" file into Portainer's volume... which defeats the purpose of managing stacks through Portainer IMO.

I see #6911 is related to this issue and looks like an internal ticket has been created? Hoping that's the case.

[wolfcry0]

I am also experiencing the same issue, referencing 'stack.env' as per the blog post results in an error:

open /data/compose/89/stack.env: no such file or directory

EE 2.14.0, docker swarm mode.

This is a bit confusing, as I'm fairly certain I am following the blog post instructions correctly.

[samdulam]

Blogpost has been updated to reflect that this currently only works for Docker Standalone and not for Docker Swarm.

[gaby]

@samdulam It also doesnt work for portainer agents, only the server.

[tamarahenson]

@gaby

I wanted to follow up on this request. I created a New Environment >> Docker Standalone >> Agent >> Create Stack >> Docker Compose >> Load stack.env >> without issue. If you are using the Docker Swarm >> Agent >> the stack.env will not work.

Thanks!

[reejosamuel]

Tried using local (docker.sock) environment and Portainer agent environment. Both give errors saying stack.env can't be found.

With portainer agent, it worked after trying the suggestion from JuniorC07. changing the path of stack.env to ../stack.env

[Klionheart]

Hello, just a couple more, hopefully useful, use cases :)

Using "stack.env" does not currently work when:

• Using Portainer CE 2.16.1
• Using a Docker Standalone environment
• Trying to deploy a stack through a git repository.

It does work, however, by using "../stack.env" as path instead.

---

Using "stack.env" on the other hand, works just fine when:

• Using Portainer CE 2.16.1
• Using a Docker Standalone environment
• Trying to deploy a stack directly with Web editor.

[tamarahenson]

@Klionheart

Thank you for the additional information. I am going to further investigate the stack.env with Git. Updates to follow.

Thanks!

[tamarahenson]

@Klionheart

Is this what you are trying to do here:

Thanks!

[marte26]

Hey, just wanted to say that I have the same problem. I'm deploying a docker-compose.yml file that sits in a subdirectory of a git repo. Trying to use stack.env as the env_file doesn't work but ../stack.env does

PS: I'm using Portainer BE 2.16.1

[Klionheart]

@Klionheart

Is this what you are trying to do here:

Thanks!

@tamarahenson the idea is to do:

where the docker-compose.yaml referenced is:

version: "3"

volumes:
    db-data:

services:
  db:
    image: postgres:12
    volumes:
      - db-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=$DB_NAME
      - POSTGRES_PASSWORD=$DB_PASSWORD
  web:
    image: healthchecks:v2.4.1
    env_file:
        - stack.env
        [...]

which should work according to https://www.portainer.io/blog/using-env-files-in-stacks-with-portainer

but results in error: error: open /data/compose/51/stack.env: no such file or directory

If I put "- ../stack.env" as the env_file in the compose.yaml, instead, it works, I guess because the stack.env file is generated as /data/compose/stack.env.

Interestingly, if I just use the same docker-compose.yaml directly in portainer (with Web Editor) it instead works with "- stack.env" as expected.

[tamarahenson]

@Klionheart

Here is my testing. I am unable to reproduce in Portainer CE or BE:

https://user-images.githubusercontent.com/20426210/202811730-49298bc8-484f-4f43-ac31-917dc51dd990.mov

This is my wordpress-env.yml https://github.com/tamarahenson/automation-scripts/blob/main/stack-code-examples/wordpress-env.yml

    env_file:
      - stack.env

This is my stack.env that resides in the same path stack-code-examples https://github.com/tamarahenson/automation-scripts/blob/main/stack-code-examples/stack.env

My mistake earlier was trying to call the stack.env in the Additional paths. Those are for .yml only.

Thanks!

[Klionheart]

Hi again @tamarahenson, I think I see the misunderstanding here.

I'm not trying to reference an existing "stack.env" inside the git repository. In fact I absolutely do not want an .env file in the git repository since it contains secrets.

The idea is to do as documented in https://www.portainer.io/blog/using-env-files-in-stacks-with-portainer where it is explained that you can add Environment variables to the local Portainer stack, and then reference them en masse by using "stack.env" as an env_file, instead of manually referencing them one by one, in the docker-compose.yaml.

This indeed works with Web editor in Docker Standalone, but not in "Repository" mode where you need to use "../stack.env" instead.

Hopefully this is more clear, thank you for your time :)

[tamarahenson]

@Klionheart

Thank you for the additional information. I figured out how to reproduce this.

[1] I changed the name of my stack.env to stack.env.bak in my repo [2] Create Stack with wordpress.yml in my repo using - stack.env [3] Upload my local stack.env [4] Received error [5] Updated my wordpress.yml to - ../stack.env [6] Deployed without error

TODO: Update our Blog with further instructions for Git Repositories

Thanks!

[Pwuts]

Tried using local (docker.sock) environment and Portainer agent environment. Both give errors saying stack.env can't be found.

With portainer agent, it worked after trying the suggestion from JuniorC07. changing the path of stack.env to ../stack.env

I am experiencing the exact opposite on Portainer 2.16.2, Docker Standalone environment connected through portainer/agent:latest:

• env_file: stack.env works
• env_file: ../stack.env -> file not found

This thread and the documentation could really use some clarification on this.

[fcaspani01]

My workaround is manually insert stack.env file inside portainer's docker volume then i updated related docker stack and now it works.

[Pwuts]

@fcaspani01 nice that it works but extremely bodgy and non-portable workaround.

[rudelm]

hi, I've also got the portainer/portainer-ee:latest container with a local docker environment controlled via /var/run/docker.sock on my Synology NAS and also encountered the problem with the stack.env parameter. If it is just stack.env it will fail, but ../stack.env will work just fine.

Stacks are also deployed via Git reference.

[koelle25]

There currently is a difference on using env_file: stack.yml between Portainer on Docker Standalone and Portainer on Docker Swarm, see also #6814.

According to this other issue it is "due to a lack of env file support in docker stack deploy".

But I think there are some big misunderstandings between using a .env file to substitute/insert values into a docker-compose.yml and using an env_file for directly inserting environment variable into a container.

It's perfectly valid and possible to do the latter. Assume a stack.yml (or docker-compose.yml, the file name does not matter) contains:

version: '3.8'

services:
  test:
    image: ubuntu
    command: env
    env_file: stack.env

And there is a stack.env file containing e.g.:

MYVAR1=something
FOO=bar

Running docker stack deploy -c stack.yml test succeeds and prints in the log (docker service logs test_test):

test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | HOSTNAME=9194cb8346cc
test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | FOO=bar
test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | MYVAR1=something
test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | HOME=/root

[towerplease]

I guess i have the same issue on a standalone portainer. I want to deploy invoiceninja, which intents you to have all configuration made outside of the docker-compose.yml-file. I have a special path-system to manage my volumes and compose-files:

/mnt/docker/compose/XYZ/docker-compose.yml, config and stuff /mnt/docker/volume/XYZ/data / sql / app

As i insert the docker-compose.yml into the webUI, i changed "env_file: env" to "env_file: /mnt/docker/compose/invoiceninja/env" but i get the error "File not found".

I tripple checked the path and used copy and paste, but it seems portainer wont find the env-file.

[pxlfrk]

I guess i have the same issue on a standalone portainer. I want to deploy invoiceninja, which intents you to have all configuration made outside of the docker-compose.yml-file. I have a special path-system to manage my volumes and compose-files:

/mnt/docker/compose/XYZ/docker-compose.yml, config and stuff /mnt/docker/volume/XYZ/data / sql / app

As i insert the docker-compose.yml into the webUI, i changed "env_file: env" to "env_file: /mnt/docker/compose/invoiceninja/env" but i get the error "File not found".

I tripple checked the path and used copy and paste, but it seems portainer wont find the env-file.

As far as i now, the referenec path should be relative to the compose-File, so in your example "env_file: .env" should work.

[darer]

For some reason I have no luck using stack.env or ../stack.env. Both result in file not found. I tried the web editor and also uploading my own compose file. Any ideas?

Portainer 2.16.1, Docker Standalone environment.

Edit:

I really need this to work as I have to run a dozen services and there are many env vars some of which are used by multiple services. I could go and list them all one by one for each service in the compose file, but maintaining them and updating them will be a nightmare.

[towerplease]

Could anyone from Portainer reproduce this behaviour and let us know if this is a bug while searching for the "env"-file with absolute Path? Would like to know if i need to reconstruct my environment or if i have to stick with my workaround. Thanks!

[darioackermann]

We are also seeing this issue and we are very certain this feature has worked before.

[Quinferris]

I am also experiencing the same issue running Community Edition 2.18.1.

I have tried using ".env" & "stack.env" and it is still throwing errors.

[janaxhell]

Hello, trying to deploy Authentik I encountered this problem as well. I have tried to copy the .env file outside /root folder, chmod g-wx .env and load it from there, but I keep getting the same error. Tried to recycle the variables using Advanced mode, but still no-go. I'm on 2.18.2 running on Armbian/OMV6 on Orange Pi 3 LTS.

[p4block]

Pretty much if any of the Host->Setup Docker Security settings are enabled, this feature won't work. It works ootb for me on docker.sock

[crosa7]

Any updates on this? It is reproducible for me in Portainer Business Edition 2.18.3

[threehappypenguins]

What fixed it for me is in the compose web editor in Portainer, I changed ALL instances of:

env_file:
      - .env

to

env_file:
      - stack.env

I am using Portainer Community Edition 2.18.3. You can't just change one .env to stack.env, or it will still throw errors. You have to change all of them.

[devtronic]

It seems Portainer itself is checking the existence of env_file entries. Since Portainer is running as a container, it can't find the files unless you add a volume to the host file.

Example solution: Your stack:

version: '3.9'

services:
  foo:
    image: foo:latest
    env_file:
      - /opt/portainer-config/my-stack-config/foo.env

Portainer stack:

# ...

  portainer:
    image: portainer/portainer-ce:2.18.4
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "9443:9443"
      - "9000:9000"
      - "8000:8000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - portainer_data:/data
      - /opt/portainer-config:/opt/portainer-config:ro # Paths must match!
# ...

Important:

• The path inside portainer MUST equal the path on the host.
• The .env file:s must exist on every single node on which the stack may run. Think about an cifs volume or something similar.

[UrsaSolar]

This bug just randomly started affecting us. We are using Docker standalone. After some quick testing, it seems that using stack.env is no longer compatible with "Enable relative path volumes" when deploying from a repository. If I disable relative path volumes, it deploys. This combination worked before without issue, but I can't even clone the stack that originally worked on now. I simply get the error that stack.env could not be found.

None of the workarounds here work for me. I tried ../stack.env, didn't work. I originally had the host filesystem mounted into /host but I tried removing that and mounting the relative path location (just /mnt) into Portainer, didn't work. I even tried manually creating the file as a dirty fix, but Portainer just deletes it and then complains it doesn't exist.

[devtronic]

None of the workarounds here work for me. I tried ../stack.env, didn't work. I originally had the host filesystem mounted into /host but I tried removing that and mounting the relative path location (just /mnt) into Portainer, didn't work. I even tried manually creating the file as a dirty fix, but Portainer just deletes it and then complains it doesn't exist.

Have you tried to mount the path identically as in my example above? It's important that the path inside the portainer service matches the path on the host machine.

[Geekimo]

Hello, Using docker swarm on the latest BE. Got the same issue about stack.env not being found in /data/compose/[...] I took a look into portainer volume, but it looks like the stack.env is not generated. I can only found the docker-compose.yml file.

[a-camacho]

Same issue here.

[rulim34]

This bug just randomly started affecting us. We are using Docker standalone. After some quick testing, it seems that using stack.env is no longer compatible with "Enable relative path volumes" when deploying from a repository. If I disable relative path volumes, it deploys. This combination worked before without issue, but I can't even clone the stack that originally worked on now. I simply get the error that stack.env could not be found.

None of the workarounds here work for me. I tried ../stack.env, didn't work. I originally had the host filesystem mounted into /host but I tried removing that and mounting the relative path location (just /mnt) into Portainer, didn't work. I even tried manually creating the file as a dirty fix, but Portainer just deletes it and then complains it doesn't exist.

I just encountered the exact same conditions here. The stack won't deployed (the stack.env not found) when I enabled the relative path volumes.

[darioackermann]

It's a bit disappointing to see this issue still being open. I am happy to provide a reproduction if needed but this is a long-term dealbreaker for us. Can anyone of Portainer Crew please comment why this is not worked on? Stacks are not unimportant...

[samdulam]

Can you give details of the stack? Include screenshots of the repo folder structure along with Stack config screen in Portainer. I'll try and look into all the different examples and come back to you with an update.

Also don't forget to mention the platform details: Docker Standalone / Swarm, OS and CPU Architecture. If you are using relative paths then details of that too.. whether that path is local disk or a NFS/CIFS mount

[UrsaSolar]

On my end this is easily reproduced using the following:

Docker Standalone, using a server & a couple agents CPU Arch: x86_64 Relative path: /mnt/portainer on local filesystem

Brand new repo with only example.yml: https://github.com/portainer/portainer/assets/81395386/c988aa42-3a7f-4823-8bf1-1d165102f630

Full portainer config screen: https://github.com/portainer/portainer/assets/81395386/8dc12d24-f044-4a28-972f-103194877433

example.yml itself contains only the following:

services:
  hello_world:
    image: hello-world
    env_file:
      - stack.env

The error: https://github.com/portainer/portainer/assets/81395386/10d641c4-414a-47f1-8554-d667eba3e2e9

Either removing the env_file entry or disabling relative paths results in a successful deployment. Combining the two results in an error. This worked previously, as I had a stack configured that way which broke not too long ago. I could no longer pull and redeploy that stack, or create a new stack with the same config.

I tried rolling back the versions to test but it resulted in some instability. I haven't had time to spin up fresh environments to play with and see if I can figure out when it broke.

[darioackermann]

When was the relative paths feature introduced? For us it seems like this

• Our original stack was created before this feature was released
• After the feature was introduced (and old stacks taking a default I guess?) things broke
• Deleting and recreating the stack under same name (with relative paths off) seems to have fixed the problem.

[samdulam]

@UrsaSolar When you don't have a stack.env file in the repo, why are you referencing it in the compose YAML? Are you using EXAMPLE_VAR in the compose for variable substitution?

[samdulam]

@darioackermann Relative paths support was introduced in 2.17 and it does not get turned on by default.

Do you also reference stack.env in the compose file but not have it in the repo? If so, why?

[Geekimo]

@UrsaSolar When you don't have a stack.env file in the repo, why are you referencing it in the compose YAML? Are you using EXAMPLE_VAR in the compose for variable substitution?

Maybe because the documentation states that this file is generated by portainer when deploying a stack ?

[samdulam]

@Geekimo Can you send me a link to that document please?