Problem setting environment variables through docker-compose

[datracka]

I have problems to create a custom DB in docker-compose using the environment variables and postgres docker image.

having this docker.compose.yml file and running docker-compose up -d

postgres:
  restart: 'true'
  image: postgres
  ports:
    - "5432:5432"
  environment:
    - POSTGRES_DB:test_db 

the container created does not contain the test_db database

But if I run the following command:

docker run -e POSTGRES_DB=test_db --name postgres-test -d -p 5432:5432 postgres

the created container DO have the test_db

I was reading the following links and everything looks good.

https://hub.docker.com/_/postgres/ https://docs.docker.com/engine/reference/run/#/env-environment-variables https://docs.docker.com/compose/environment-variables/

Sure I am missing something because it should be straightforward. But what?

thank for your time and help!

[mrafayaleem]

I suspect this is happening because you have a previous image of this in your docker where you ran docker-compose up without providing the envs and hence it is not able to overwrite. Check and delete postgres image by doing docker ps -a and docker rm <image> respectively.

[datracka]

Hi,

it did not work :( I removed all the images in my local machine and regardless still not working.

Below the log output when I run $ docker-compose up

postgres_1  | The files belonging to this database system will be owned by user "postgres".
postgres_1  | This user must also own the server process.
postgres_1  | 
postgres_1  | The database cluster will be initialized with locale "en_US.utf8".
postgres_1  | The default database encoding has accordingly been set to "UTF8".
postgres_1  | The default text search configuration will be set to "english".
postgres_1  | 
postgres_1  | Data page checksums are disabled.
postgres_1  | 
postgres_1  | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1  | creating subdirectories ... ok
postgres_1  | selecting default max_connections ... 100
postgres_1  | selecting default shared_buffers ... 128MB
postgres_1  | selecting dynamic shared memory implementation ... posix
postgres_1  | creating configuration files ... ok
postgres_1  | running bootstrap script ... ok
postgres_1  | performing post-bootstrap initialization ... ok
postgres_1  | syncing data to disk ... ok
postgres_1  | 
postgres_1  | WARNING: enabling "trust" authentication for local connections
postgres_1  | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1  | --auth-local and --auth-host, the next time you run initdb.
postgres_1  | 
postgres_1  | Success. You can now start the database server using:
postgres_1  | 
postgres_1  |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1  | 
postgres_1  | ****************************************************
postgres_1  | WARNING: No password has been set for the database.
postgres_1  |          This will allow anyone with access to the
postgres_1  |          Postgres port to access your database. In
postgres_1  |          Docker's default configuration, this is
postgres_1  |          effectively any other container on the same
postgres_1  |          system.
postgres_1  | 
postgres_1  |          Use "-e POSTGRES_PASSWORD=password" to set
postgres_1  |          it in "docker run".
postgres_1  | ****************************************************
postgres_1  | waiting for server to start....LOG:  database system was shut down at 2016-10-17 08:45:42 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  database system is ready to accept connections
postgres_1  | LOG:  autovacuum launcher started
postgres_1  |  done
postgres_1  | server started
postgres_1  | ALTER ROLE
postgres_1  | 
postgres_1  | 
postgres_1  | /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1  | 
postgres_1  | LOG:  received fast shutdown request
postgres_1  | LOG:  aborting any active transactions
postgres_1  | LOG:  autovacuum launcher shutting down
postgres_1  | LOG:  shutting down
postgres_1  | waiting for server to shut down....LOG:  database system is shut down
postgres_1  |  done
postgres_1  | server stopped
postgres_1  | 
postgres_1  | PostgreSQL init process complete; ready for start up.
postgres_1  | 
postgres_1  | LOG:  database system was shut down at 2016-10-17 08:45:43 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  database system is ready to accept connections
postgres_1  | LOG:  autovacuum launcher started

has something to do this line? /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* I am pointing in this direction: I assume my docker configuration should be tweak in some way... (I am using Docker for Mac Version 1.12.1 (build: 12133))

at the moment and as workaround I create the DB / user / pass manually.

[mrafayaleem]

I still don't get it but I am using the following docker-compose.yml and it works perfectly fine. I was in a similar problem that you have mentioned and the trick was to just remove the previous docker image and data dir and re-run the container from scratch.

version: '2'
services:
  postgres:
    image: postgres:9.3
    ports:
    - "5432:5432"
    environment:
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
      - POSTGRES_DB=test_db
    volumes:
      - ./data/postgres:/var/lib/postgresql/data

[datracka]

your script works like a charm! Still I do not get exactly what I was doing wrong (maybe it was the version:'2' or nest the services under 'services' tag..), but it does not mind, I move forward. Thanks for your help!

[Chrisissorry]

Had the exact same problem. Removed all images and deleted the data directory as @mrafayaleem suggested - works!

[yosifkit]

Yeah, the problem is not the existing images on your host, but it stems from the volume (or bind mount directory) being already initialized after your first start. The postgres user, and database creation only happens on the first start (ie, /var/lib/postgresql/data must not already contain database files).

Here is an example flow

1. declare volume for my image via compose: ./data/postgres:/var/lib/postgresql/data
2. set env in yaml: POSTGRES_PASSWORD=test
3. start containers docker-compose up -d
4. decide to add database to postgres env POSTGRES_DB=test_db or change the password
5. restart my containers docker-compose up -d
6. database test_db does not exist or password is unchanged :cry:
   1. db files must be removed in order for postres entrypoint to re-initialize: docker-compose stop; sudo rm -rf ./data/postgres/
   2. now it can be restarted docker-compose up -d :smiley:

If you do not declare a volume mount point, then the VOLUME declared in the postgres image will apply and docker will create and manage the directory independent of the life-cycle of the container. This gets more complicated when using compose, since it will keep the volume to re-use later even when you docker-compose rm -f all of your running containers.

An example without a bind mounted volume:

1. say we do steps 2-5 above
2. we still have the same problem of the test_db database not existing (or the password not changing) since the data from the volume still exists :disappointed:
   1. so, get rid of all containers and their volumes docker-compose rm -fv
   2. or docker-compose rm -fv postgres to get rid of just the postres service and its volumes
   3. now we can start up a new postgres container with a new empty volume docker-compose up -d :smiley:

You can see what volumes you have on your host by doing a docker volume ls (bind mounts will not show up in this list). There is currently no easy way to see what containers are attached to a volume, so you would have to docker inspect the container to see which ones are attached to it.

If you want to clean up all local volumes that are not attached to containers (WARNING this could delete important data :astonished:): docker volume ls | awk '$1 == "local" { print $2 }' | xargs --no-run-if-empty docker volume rm. On my development machine I usually precede this by removing stopped containers.

Also, in case it is not obvious, do not delete your postres data directory or volume if you have important data stored there. :scream:

[erikj]

I do not know for sure if it was the root cause, but the syntax used to specify the test_db environment variable appears to be incorrect and will not set POSTGRES_DB:

  environment:
    - POSTGRES_DB:test_db 

You can either specify it as a hash (no -):

  environment:
    POSTGRES_DB:test_db 

Or as an array, -, w/ = assignment rather than key:value

  environment:
    - POSTGRES_DB=test_db 

[i-pip]

This issue is still not resolved for me. I tried removing the images after shutdown with docker-compose down --rmi all then deleted my data/postgres folder, confirmed that there are no volumes still attached with docker volume ls and even tried --force-recreate with docker-compose up --force-recreate. docker-compose still ignores the environment variables in my docker-compose.yml which looks like so:

services:
  web:
    restart: always
    build: . 
    ports:
      - "3000:3000"
    links:
      - db
  db:
    image: postgres
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=someuser
      - POSTGRES_PASSWORD=s0me-p4sswd
      - POSTGRES_DB=test-db
      - POSTGRES_PORT=5432
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    restart: always

I still get the following output

db_1   | syncing data to disk ... ok
db_1   |
db_1   | Success. You can now start the database server using:
db_1   |
db_1   |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1   |
db_1   |
db_1   | WARNING: enabling "trust" authentication for local connections
db_1   | You can change this by editing pg_hba.conf or using the option -A, or
db_1   | --auth-local and --auth-host, the next time you run initdb.
db_1   | waiting for server to start....2018-02-16 05:10:10.335 UTC [40] LOG:  listening on IPv4 address "127.0.0.1", port 5432
db_1   | 2018-02-16 05:10:10.335 UTC [40] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
db_1   | 2018-02-16 05:10:10.335 UTC [40] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
db_1   | 2018-02-16 05:10:10.338 UTC [40] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1   | 2018-02-16 05:10:10.397 UTC [41] LOG:  database system was shut down at 2018-02-16 05:10:08 UTC
db_1   | 2018-02-16 05:10:10.420 UTC [40] LOG:  database system is ready to accept connections
db_1   |  done
db_1   | server started
db_1   | CREATE DATABASE
db_1   |
db_1   | CREATE ROLE
db_1   |
db_1   |
db_1   | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1   |
db_1   | waiting for server to shut down...2018-02-16 05:10:12.424 UTC [40] LOG:  received fast shutdown request
db_1   | .2018-02-16 05:10:12.426 UTC [40] LOG:  aborting any active transactions
db_1   | 2018-02-16 05:10:12.429 UTC [40] LOG:  worker process: logical replication launcher (PID 47) exited with exit code 1
db_1   | 2018-02-16 05:10:12.430 UTC [42] LOG:  shutting down
db_1   | 2018-02-16 05:10:12.490 UTC [40] LOG:  database system is shut down
db_1   |  done
db_1   | server stopped
db_1   |
db_1   | PostgreSQL init process complete; ready for start up.
db_1   |
db_1   | 2018-02-16 05:10:12.549 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1   | 2018-02-16 05:10:12.549 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1   | 2018-02-16 05:10:12.552 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1   | 2018-02-16 05:10:12.636 UTC [67] LOG:  database system was shut down at 2018-02-16 05:10:12 UTC
db_1   | 2018-02-16 05:10:12.667 UTC [1] LOG:  database system is ready to accept connections```

[yosifkit]

@doc-phily, the output of your logs look correct. The CREATE DATABASE and CREATE ROLE output is where postgres is logging the fact that the entrypoint script created them. What problems are you seeing?

On a side note, POSTGRES_PORT isn't doing anything, since it is not a variable the entrypoint scripts looks for (docs).

[auraz]

Had this issue when postgres data volume was in /var/.. where my user had no access right. After moving it to home directory, it works.

[HouWangSFDC]

encounter same issue, @auraz your suggestion works perfectly! many thanks

[tianon]

Closing given that the original issue is resolved (and this issue ought to be plenty searchable by now). :+1:

[kgawrys]

Had the same issue. In my case it was incorrectly defined volume:

I had volume definition in docker-compose.yml: volumes: postgres-data:

I defined the volume for postgres like this: ` volumes:

• '~/docker-volumes/postgres-data:/var/lib/postgresql/data' while I should define it like below:
  volumes:
• 'postgres-data:/var/lib/postgresql/data' `

Because of the first version the volume was not removed during docker volume rm and because of that I couldn't update password with POSTGRES_PASSWORD, even after doing docker system prune -a (care with this command as it removes everything docker-related) :)

[RomarQ]

Fix: https://github.com/nextcloud/docker/issues/345#issuecomment-398037012

[BLockhartCC]

There may be an older instance of the image. Be sure to run docker-compose build.

[wansiedler]

Yeah, the problem is not the existing images on your host, but it stems from the volume (or bind mount directory) being already initialized after your first start. The postgres user, and database creation only happens on the first start (ie, /var/lib/postgresql/data must not already contain database files).

Here is an example flow

1. declare volume for my image via compose: ./data/postgres:/var/lib/postgresql/data
2. set env in yaml: POSTGRES_PASSWORD=test
3. start containers docker-compose up -d
4. decide to add database to postgres env POSTGRES_DB=test_db or change the password
5. restart my containers docker-compose up -d

6. database test_db does not exist or password is unchanged 😢

   1. db files must be removed in order for postres entrypoint to re-initialize: docker-compose stop; sudo rm -rf ./data/postgres/
   2. now it can be restarted docker-compose up -d 😃

If you do not declare a volume mount point, then the VOLUME declared in the postgres image will apply and docker will create and manage the directory independent of the life-cycle of the container. This gets more complicated when using compose, since it will keep the volume to re-use later even when you docker-compose rm -f all of your running containers.

An example without a bind mounted volume:

1. say we do steps 2-5 above

2. we still have the same problem of the test_db database not existing (or the password not changing) since the data from the volume still exists 😞

   1. so, get rid of all containers and their volumes docker-compose rm -fv
   2. or docker-compose rm -fv postgres to get rid of just the postres service and its volumes
   3. now we can start up a new postgres container with a new empty volume docker-compose up -d 😃

You can see what volumes you have on your host by doing a docker volume ls (bind mounts will not show up in this list). There is currently no easy way to see what containers are attached to a volume, so you would have to docker inspect the container to see which ones are attached to it.

If you want to clean up all local volumes that are not attached to containers (WARNING this could delete important data 😲): docker volume ls | awk '$1 == "local" { print $2 }' | xargs --no-run-if-empty docker volume rm. On my development machine I usually precede this by removing stopped containers.

Also, in case it is not obvious, do not delete your postres data directory or volume if you have important data stored there. 😱

WORKED! =)

[escalonn]

anybody know a convenient way to have postgres run a script on startup even if the data directory is not empty? i have an idempotent migration script i just want to run any time the container starts.

[thomascrha]

anybody know a convenient way to have postgres run a script on startup even if the data directory is not empty? i have an idempotent migration script i just want to run any time the container starts.

@escalonn Did you find a way to do this ?

[escalonn]

@thomascrha nope. i don't think it exists so when i get back to that project i plan to just psql it from another container.

[djheru]

FYI, was helping a teammate with this issue today. It turns out they had a local installation of PostgreSQL running, and that was causing the issue. Once we stopped the (brew packaged) service with brew services stop postgresql, docker-compose worked as expected, setting the correct user and database name from the environment variables.

[falk-stefan]

For me adding --renew-anon-volumes did the trick (for now).

[lachezar]

--renew-anon-volumes seems to fix it for me... I also had a "." (dot) in the name if the parent folder and for some reason after renaming it to something without a dot in its name it started working. It could've been the dot, but it could have been the act of renaming it... idk 🤷

[gzlock]

I'm stuck in this issue and can't solve it.

Env:

• win11
• docker desktop v4.30.0
• wsl2 + Ubuntu 22.04.4 LTS

Tried:

• Docker image: postgres:15.4-alpine , postgres:15-alpine, postgres:latest
• removed all volumes.
• Reseted docker to factory defaults.

I maked sure to delete all volumes and try to run docker-compose up -d and docker-compose down -v It always creates and uses the username "postgres" (first line of the log) I tried to login with username test or postgres and password test123 but failed. so I don't know what's the password now.

docker-compose.yml

services:
  db:
    restart: always
    image: postgres:15.4-alpine
    container_name: postgres
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test123
    expose:
      - 5432
    ports:
      - "5432:5432"

Log:

2024-05-18 06:48:05 The files belonging to this database system will be owned by user "postgres".
2024-05-18 06:48:05 This user must also own the server process.
2024-05-18 06:48:05 
2024-05-18 06:48:05 The database cluster will be initialized with locale "en_US.utf8".
2024-05-18 06:48:05 The default database encoding has accordingly been set to "UTF8".
2024-05-18 06:48:05 The default text search configuration will be set to "english".
2024-05-18 06:48:05 
2024-05-18 06:48:05 Data page checksums are disabled.
2024-05-18 06:48:05 
2024-05-18 06:48:05 fixing permissions on existing directory /var/lib/postgresql/data ... ok
2024-05-18 06:48:05 creating subdirectories ... ok
2024-05-18 06:48:05 selecting dynamic shared memory implementation ... posix
2024-05-18 06:48:05 selecting default max_connections ... 100
2024-05-18 06:48:05 selecting default shared_buffers ... 128MB
2024-05-18 06:48:05 selecting default time zone ... UTC
2024-05-18 06:48:05 creating configuration files ... ok
2024-05-18 06:48:05 running bootstrap script ... ok
2024-05-18 06:48:05 performing post-bootstrap initialization ... ok
2024-05-18 06:48:06 syncing data to disk ... ok
2024-05-18 06:48:06 
2024-05-18 06:48:06 
2024-05-18 06:48:06 Success. You can now start the database server using:
2024-05-18 06:48:06 
2024-05-18 06:48:06     pg_ctl -D /var/lib/postgresql/data -l logfile start
2024-05-18 06:48:06 
2024-05-18 06:48:06 waiting for server to start....2024-05-17 22:48:06.235 UTC [36] LOG:  starting PostgreSQL 15.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
2024-05-18 06:48:06 2024-05-17 22:48:06.239 UTC [36] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-05-18 06:48:06 2024-05-17 22:48:06.253 UTC [39] LOG:  database system was shut down at 2024-05-17 22:48:05 UTC
2024-05-18 06:48:06 2024-05-17 22:48:06.258 UTC [36] LOG:  database system is ready to accept connections
2024-05-18 06:48:06  done
2024-05-18 06:48:06 server started
2024-05-18 06:48:06 CREATE DATABASE
2024-05-18 06:48:06 
2024-05-18 06:48:06 
2024-05-18 06:48:06 /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
2024-05-18 06:48:06 
2024-05-18 06:48:06 waiting for server to shut down...2024-05-17 22:48:06.357 UTC [36] LOG:  received fast shutdown request
2024-05-18 06:48:06 .2024-05-17 22:48:06.361 UTC [36] LOG:  aborting any active transactions
2024-05-18 06:48:06 2024-05-17 22:48:06.362 UTC [36] LOG:  background worker "logical replication launcher" (PID 42) exited with exit code 1
2024-05-18 06:48:06 2024-05-17 22:48:06.362 UTC [37] LOG:  shutting down
2024-05-18 06:48:06 2024-05-17 22:48:06.366 UTC [37] LOG:  checkpoint starting: shutdown immediate
2024-05-18 06:48:06 2024-05-17 22:48:06.541 UTC [37] LOG:  checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.016 s, sync=0.144 s, total=0.179 s; sync files=301, longest=0.005 s, average=0.001 s; distance=4223 kB, estimate=4223 kB
2024-05-18 06:48:06 2024-05-17 22:48:06.544 UTC [36] LOG:  database system is shut down
2024-05-18 06:48:05 sh: locale: not found
2024-05-18 06:48:05 2024-05-17 22:48:05.420 UTC [30] WARNING:  no usable system locales were found
2024-05-18 06:48:06 initdb: warning: enabling "trust" authentication for local connections
2024-05-18 06:48:06 initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
2024-05-18 06:48:06 2024-05-17 22:48:06.577 UTC [1] LOG:  starting PostgreSQL 15.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
2024-05-18 06:48:06 2024-05-17 22:48:06.577 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-05-18 06:48:06 2024-05-17 22:48:06.577 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-05-18 06:48:06 2024-05-17 22:48:06.586 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-05-18 06:48:06 2024-05-17 22:48:06.595 UTC [52] LOG:  database system was shut down at 2024-05-17 22:48:06 UTC
2024-05-18 06:48:06 2024-05-17 22:48:06.601 UTC [1] LOG:  database system is ready to accept connections
2024-05-18 06:48:06  done
2024-05-18 06:48:06 server stopped
2024-05-18 06:48:06 
2024-05-18 06:48:06 PostgreSQL init process complete; ready for start up.
2024-05-18 06:48:06