search

docker / for-win

Bug reports for Docker Desktop for Windows
https://www.docker.com/products/docker#/windows
1.85k stars 285 forks source link

Mounting data volume for Postgres in docker for Windows doesn't work #445

Closed vrindanayak closed 7 years ago

vrindanayak commented 7 years ago

Expected behavior

Postgres should start and mount the correct windows directory as a data volume

Actual behavior

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
initdb: could not fsync file "/var/lib/postgresql/data/base/1": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/base/12406": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/base/12407": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/base": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/global": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_clog": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_commit_ts": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_dynshmem": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_logical/mappings": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_logical/snapshots": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_logical": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_multixact/members": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_multixact/offsets": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_multixact": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_notify": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_replslot": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_serial": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_snapshots": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_stat": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_stat_tmp": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_subtrans": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_tblspc": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_twophase": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_xlog/archive_status": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_xlog": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_tblspc": Invalid argument

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....FATAL:  data directory "/var/lib/postgresql/data" has wrong ownership
HINT:  The server must be started by the user that owns the data directory.
pg_ctl: could not start server
Examine the log output.
 stopped waiting

Information

docker-compose.yml file : 

version: "2"
services:
  slapd:
    image: dcm4che/slapd-dcm4chee:2.4.40-8.1
    ports:
      - "389:389"
    env_file: docker-compose.env
    volumes:
      - //c/Users/<user>/var/local/dcm4chee-arc/ldap:/var/lib/ldap
      - //c/Users/<user>/var/local/dcm4chee-arc/slapd.d:/etc/ldap/slapd.d
  postgres:
    image: dcm4che/postgres-dcm4chee:9.6-8
    ports:
      - "5432:5432"
    env_file: docker-compose.env
    volumes:
      - //c/Users/<user>/var/local/dcm4chee-arc/db:/var/lib/postgresql/data
  dcm4chee-arc:
    image: dcm4che/dcm4chee-arc-psql:5.8.1
    ports:
      - "8080:8080"
      - "9990:9990"
      - "11112:11112"
      - "2575:2575"
    env_file: docker-compose.env
    environment:
      WILDFLY_CHOWN: /opt/wildfly/standalone /storage
      WILDFLY_WAIT_FOR: ldap:389 db:5432
      AUTH_SERVER_URL: /auth
      JAVA_OPTS: "-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true"
    links:
      - slapd:ldap
      - postgres:db
    volumes:
      - //c/Users/<user>/var/local/dcm4chee-arc/storage:/storage
      - //c/Users/<user>/var/local/dcm4chee-arc/wildfly:/opt/wildfly/standalone

Note : Substituted <user> with my user

docker-compose.env file

LDAP_BASE_DN=dc=dcm4che,dc=org
LDAP_ORGANISATION=dcm4che.org
LDAP_ROOTPASS=secret
LDAP_CONFIGPASS=secret
ARCHIVE_DEVICE_NAME=dcm4chee-arc
AE_TITLE=DCM4CHEE
DICOM_HOST=localhost
DICOM_PORT=11112
HL7_PORT=2575
STORAGE_DIR=/storage/fs1
POSTGRES_DB=pacsdb
POSTGRES_USER=pacs
POSTGRES_PASSWORD=pacs
WILDFLY_ADMIN_USER=admin
WILDFLY_ADMIN_PASSWORD=admin
KEYCLOAK_ADMIN_USER=admin
KEYCLOAK_ADMIN_PASSWORD=admin

Steps to reproduce the behavior

  1. Share Drive using Docker for Windows->Settings->Shared Drive -> C:
  2. Use docker-compose up -d

Similar issue was already reported on https://github.com/docker/docker/issues/25908 but was redirected here to be checked/opened on https://github.com/docker/for-win/issues I have searched for this issue in Open and Closed issues list, but couldn't find any.

I'm using

friism commented 7 years ago

I think this is a duplicate of https://github.com/docker/for-win/issues/39

For the time being, this won't work. You should either just use in-container storage or use a volume not mounted from your Windows host.

Is there a particular reason you want the data to be on the windows host?

colonelz commented 7 years ago

Tried it with this approach: https://forums.docker.com/t/trying-to-get-postgres-to-work-on-persistent-windows-mount-two-issues/12456/5

command line

docker volume create --name postgres-data-volume -d local

docker-compose.yml

... ### PostgreSQL Container ####################################

postgres: build: ./postgres volumes: - postgres-data-volume:/var/lib/postgresql/data ports: - "${POSTGRES_PORT}:5432" environment: - POSTGRES_DB=${POSTGRES_DB} - POSTGRES_USER=${POSTGRES_USER} - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} networks: - backend

### PostgreSQL PostGis Container ############################

postgres-postgis: build: ./postgres-postgis volumes: - postgres-data-volume:/var/lib/postgresql/data ports: - "${POSTGRES_PORT}:5432" environment: - POSTGRES_DB=${POSTGRES_DB} - POSTGRES_USER=${POSTGRES_USER} - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} networks: - backend ...

### Volumes Setup #############################################

volumes: mysql: driver: "local"

...

postgres-data-volume: external: true

redfellow commented 6 years ago

also affected

darkguy2008 commented 6 years ago

Same here.

Ah come on, so you need a specific reason why to support a core Docker functionality? Like, well, I expect it to work with storing the data in a volume instead of being stuck in some random volume that I can't move to another server due to cryptic file paths and so I want my docker data + config to stay in the same folder so it's easy for me to backup, like a good DevOp would do? Holy Christ.

If MariaDB can do it, why Postgre can't? I don't want my data to be wiped out when I turn off the container.

+1 to this, it needs to be fixed.

MatthewLymer commented 6 years ago

Same issue, luckily I'm just picking my database now.

flavienbwk commented 6 years ago

Same issue. Apparently no solutions for 1 year now...

redfellow commented 6 years ago

Docker & Production ready.

That's two separate products.

Greifenkralle commented 6 years ago

Same problem.

flavienbwk commented 6 years ago

I've solved the problem by creating a new volume like that : docker volume create postgres_database

And then you link it like that :

services:
    postgres:
        restart: always
        volumes:
            - postgres_database:/var/lib/postgresql/data:Z

volumes:
    postgres_database:
        external: true
Greifenkralle commented 6 years ago

flavienbwk: Your solution works. :) Thank you.

flavienbwk commented 6 years ago

Perfect!

Cugap commented 5 years ago

This work but this is bad solution, global docker volumes creating inside of docker VM on Windows and MAC, what make pack of problems, such as no option to modify or inspect data of volumes.

flavienbwk commented 5 years ago

@Cugap Maybe a bad solution but there's no proposed solution by the Docker team, that has closed the issue whereas it is not solved. Maybe they should do something ?

marcospgp commented 5 years ago

Please consider reopening!

Cugap commented 5 years ago

@friism pls reopen cause problem is still active, and maintaining of containerized DB is a real problem.

friism commented 5 years ago

ping @carlfischer1

carlfischer1 commented 5 years ago

ping @gtardif

beeplin commented 5 years ago

this is really frustrating. Please reopen this issue.

getsomebread commented 5 years ago

Local development with Docker Desktop for Windows as a single-node swarm is an important step to developing apps to deploy to Docker EE for us, as it allows us to create local overlays, networks and (almost) volumes and deploy stacks locally with 'docker stack deploy' + compose file. Being able to map a Windows folder to a volume (bind mount), with 'docker volume create' is quite important to replicate Docker EE functionality. Our development on Linux and macOS allows this, but many developers have Win10 laptops and this is a blocker.

Sure, we can create volumes and use them, but getting data into them for development is a bugger. We need to create a volume, then bring up a container that mounts it and then 'docker cp' our data into it or some other esoteric usage of rsync, etc. Aside from the inconvenience, it is inconsistent with how our macOS and Linux users work. PLEASE make this a priority.

ta32 commented 5 years ago

Hi am running Linux containers using LCOW https://blogs.msdn.microsoft.com/premier_developer/2018/04/20/running-docker-windows-and-linux-containers-simultaneously/

when i try and run the official postgres container i get this issue, is it related or should i open another issue?

fixing permissions on existing directory /var/lib/postgresql/data ... initdb: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted

bradfox2 commented 5 years ago

How has it really been almost 2 years with no resolution to this issue? Data stuck in an external volume, buried within a Hyper V VM is not acceptable.

BlowaXD commented 5 years ago

No news ?

bashallc commented 5 years ago

or updates?

newza-fullmetal commented 5 years ago

Same problem. Docker version 3 win 10 family. I used the workaroud but not really fan of it

NikolaosWakem commented 5 years ago

Still a problem today

nitinthewiz commented 5 years ago

Just mucking around with docker toolbox on Windows 10 Home and yeah, still a problem. This has been a problem since over 3 years now and clearly won’t ever get fixed. Sad to see that the supposed docker support in Windows is just a sham for any serious work. Oh well.

As for the why - I’ve got a ridiculously pumped up Windows machine at home that I use for gaming. Why wouldn’t I want to run applications on it too? Docker is a good solution for that, except if it keeps giving me trouble, I’ll go for bitnami instead.

SuperSandro2000 commented 5 years ago

@nitinthewiz Postgres is to picky. Windows does not support linux filesystem permissions and someone would need to write a custom driver. If you go for Bitnami you are changing basically the entire software.

MatthewLymer commented 5 years ago

Windows Subsystem for Linux seems to support Linux permissions on NTFS.

flavienbwk commented 5 years ago

@MatthewLymer did you manage to install Docker on WSL ?

MatthewLymer commented 5 years ago

No, nor am I likely to attempt it, just saying that products are able to represent Linux permissions on Windows systems.

SuperSandro2000 commented 5 years ago

@flavienbwk that won't work

@MatthewLymer if you touch the WSL directory with normal windows you totally break it

nitinthewiz commented 5 years ago

@SuperSandro2000 indeed bitnami would mean changing my entire stack. The objective is fast and easy deployment, not development, so the stack doesn't matter if it works. For now, I've got all the cogs turning and the setup is working. For a bit, I had a problem where I didn't know how to add plugins to tt-rss (which is what I'm trying to deploy) because it's inside docker, but I logged in using docker -it container_name bash and was able to use git to pull the plugin files I needed.

Also, I'm kinda trusting postgres to just work hands off from this point onward. If it starts eating up all my storage or something, I'll think about touching the setup.

SuperSandro2000 commented 5 years ago

You need to mount the plugin in a volume of it gets deleted when you restart.

Postgress works on Linux like a charm

Iristyle commented 5 years ago

FYI the native Linux Postgres 9.6 container will at least start under Windows LCOW mode now, given a few conditions:

There are some instructions for setting this all up that I've been maintaining at https://github.com/puppetlabs/pupperware/blob/master/README-windows.md

marcinius commented 5 years ago

Still no resolution to this after all this time. Isn't this just wonderful.

SuperSandro2000 commented 5 years ago

This will probably be solved when Docker moves to WSL2

saeed349 commented 5 years ago

Looks like all the Windows 10 home users are screwed

SuperSandro2000 commented 5 years ago

Looks like all the Windows 10 home users are screwed

They aren't. Hyper-V should be available on Home, too.

jithurjacob commented 5 years ago

Waiting for this!

gbritton1 commented 5 years ago

Dunno if WSL2 will solve the problem or not.  What I want is to get the postgres container to use a Windows folder (not in a vhd) for its persistent storage.  That way I can easily move that folder to another machine, expose it to another container, even load postgresql native on my windows box and use the same folder (not all the same time, of course!) 

 

I'm not sure if this is a Docker or Postgresql problem (or both?) but I'd sure like to see it solved.

SuperSandro2000 commented 5 years ago

@gbritton1 won't happen as postgres is very picky about it's permission and NTFS dies nur support such permissions. Either there is a windows port or you just use Linux.

gbritton1 commented 5 years ago

Is there a way to tell docker to set the ownership of the volume in the container to the user postgres instead of root? That might get around the problem.

Iristyle commented 5 years ago

Again, see my comment at https://github.com/docker/for-win/issues/445#issuecomment-480549220

Linux Postgres works just fine under LCOW, assuming you meet the pre-requisites as I've described (it had startup permission issues under RS4, but once moving to RS5, those issues went away). We've been running it in CI for months as part of our testing of a full Linux stack under LCOW.

There's a recent CI run at https://dev.azure.com/puppetlabs/pupperware/_build/results?buildId=3319 where you can see the logs emitted from the Postgres container.

WSL2 is not really relevant to the discussion here. If you're using LCOW, you're already running a LinuxKit derived Linux kernel from https://github.com/linuxkit/lcow (and prior to LCOW there was the MobyLinuxVM)

The relevant section of the docker-compose.yml for Postgres is at https://github.com/puppetlabs/pupperware/blob/master/docker-compose.yml#L30-L45

gbritton1 commented 5 years ago

Right, so getting back to the original problem. I want Postgres in LCOW to use a Windows folder for the database files. This will give me the portability I'm looking for. (Of course I can dump the db and reload it elsewhere but I'm hoping to avoid that). AFAIK, there is at present no way to do that (but boy, would I like to be wrong!)

The sample docker-compose.yml files don't really address this, or I just can't find it.

Iristyle commented 5 years ago

The volume definitions for Postgres is at line https://github.com/puppetlabs/pupperware/blob/master/docker-compose.yml#L39 in the docker-compose.yml I linked to.

- ${VOLUME_ROOT:-.}/volumes/puppetdb-postgres/data:/var/lib/postgresql/data

The left side of the : is the Windows path and the right side is the path inside the container. We have the Windows path parameterized with the VOLUME_ROOT environment variable, but that's not necessary if you have a fixed path. You could do something like:

- c:/windows/temp/postgres-data:/var/lib/postgresql/data
gbritton1 commented 5 years ago

That results in the "wrong ownership" message in the top post.

Iristyle commented 5 years ago

@gbritton you haven't confirmed what build of Windows, Docker or the LCOW kernel you're running, so I suspect you aren't using the right combination of software.

You must satisfy minimum versions of all 3 for it to work.

We've been running Linux Postgres on a Windows host for 4+ months now. If you look at the Azure DevOps logs I linked, there's info about the host we're using and logs showing Postgres databases being created, SQL scripts running against them, etc etc. All data is stored on an NTFS partition volume mapped into the container.

gbritton1 commented 5 years ago

Windows 10 Pro: 1903, build 18362.239 Docker: version 18.09.2, build 6247962 (released Feb 2019, well after Win Server 2019 (Oct 2018)) -- experimental features enabled Linux: 4.9.125-linuxkit Postgres:latest 11.4 (Debian 11.4-1.pgdg90+1)

I see your log does not contain the failure the rest of us are hitting. Don't know what your magic foo is. Just wish I had it!

Iristyle commented 5 years ago

@gbritton1 looks like you've made some assumptions about your stack being the same that might not be true.

gbritton1 commented 5 years ago

Only real assumption is that this should wort with the stable docker branch. If it works with that eventually I'll be happy.

On Sat, Jul 27, 2019, 3:31 PM Ethan J. Brown notifications@github.com wrote:

@gbritton1 https://github.com/gbritton1 looks like you've made some assumptions about your stack being the same that might not be true.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/docker/for-win/issues/445?email_source=notifications&email_token=AC5R66ABCGQP2N6GWUSM4WLQBSPBPA5CNFSM4C6F2NK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26RI3I#issuecomment-515708013, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5R66D6CN7HDVWJKXR6BD3QBSPBPANCNFSM4C6F2NKQ .