Postgresql problem with mounted volume for data

[seantanly]

I believe this is the same problem with https://github.com/Parallels/docker-machine-parallels/issues/28 and it works on Docker for Mac (Hyperkit) but not via docker-machine-parallels.

I have just purchased Parallels Pro for the reasons it has much better IO performance vs (free) Docker for Mac. prl_fs being unable to fsync looks like a big showstopper. Is fsync for prl_fs going to be worked on?

2017-11-30 04:34:40.135 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2017-11-30 04:34:40.135 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2017-11-30 04:34:40.137 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2017-11-30 04:34:40.170 UTC [21] LOG:  database system was interrupted; last known up at 2017-11-28 11:55:57 UTC
2017-11-30 04:34:40.845 UTC [21] LOG:  could not fsync file "./base/1": Invalid argument
2017-11-30 04:34:40.923 UTC [21] LOG:  could not fsync file "./base/106898": Invalid argument
2017-11-30 04:34:40.998 UTC [21] LOG:  could not fsync file "./base/108500": Invalid argument
2017-11-30 04:34:41.068 UTC [21] LOG:  could not fsync file "./base/110098": Invalid argument
2017-11-30 04:34:41.136 UTC [21] LOG:  could not fsync file "./base/126482": Invalid argument
2017-11-30 04:34:41.205 UTC [21] LOG:  could not fsync file "./base/12993": Invalid argument
2017-11-30 04:34:41.290 UTC [21] LOG:  could not fsync file "./base/12994": Invalid argument
2017-11-30 04:34:41.290 UTC [21] LOG:  could not fsync file "./base": Invalid argument
2017-11-30 04:34:41.308 UTC [21] LOG:  could not fsync file "./global": Invalid argument

[romankulikov]

fsync should not return errors on prl_fs. Can you please help me with investigation by providing strace output of postgresql process when issue is reproduced?

[seantanly]

I'm not sure how to do a strace, but here's the steps to reproduce the errors.

    HOST_DATA_PATH=~/docker/postgres/data

    # Run instance to initialize DB
    docker run \
      -d \
      --rm \
      -p 5432:5432 \
      --name postgres \
      -e POSTGRES_USER=postgres \
      -e POSTGRES_PASSWORD=postgres \
      -e PGDATA=/var/lib/postgresql/data/pgdata \
      postgres:10.1

    # Wait for DB to initialize
    sleep 15

    # Copy out initialized DB data
    mkdir -p $HOST_DATA_PATH
    docker cp postgres:/var/lib/postgresql/data/pgdata $HOST_DATA_PATH/pgdata

    # verify DB data is copied
    ls -al $HOST_DATA_PATH/pgdata

    # Stop 1st instance which is no longer needed
    docker stop postgres

    # Run actual workhorse instance and see the error
    docker run \
      --rm \
      --name postgres \
      -p 5432:5432 \
      -e POSTGRES_USER=postgres \
      -e POSTGRES_PASSWORD=postgres \
      -e PGDATA=/var/lib/postgresql/data/pgdata \
      -v $HOST_DATA_PATH:/var/lib/postgresql/data \
      postgres:10.1

[romankulikov]

Found it. This issue is caused by the absence of fsync operation support for directories in prl_fs.

[seantanly]

Cool, will the fsync operation support be added soon? Does that need upgrading of Parallels to next patch version or a simple update of docker-machine-parallels?

[romankulikov]

Neither docker-machine-parallels nor Parallels Desktop. Issue should be fixed in boot2docker image really. Will try to prepare pull request with workaround.

[seantanly]

Thanks, can you ref the boot2docker PR with this issue? Makes it easier to follow.

Hoping to see this issue resolved soon. :)

[romankulikov]

Issues is fixed in boot2docker 17.12.0-ce-rc4.

[seantanly]

Thank you, this issue is resolved with the release of boot2docker 17.12.0-ce a day ago.

[iby]

@romankulikov I'm running into what might be the same issue with:

docker run -d --name postgres -e POSTGRES_PASSWORD=postgres -v /Users/ianbytchek/Documents/postgres:/var/lib/postgresql/data -p 5432:5432 postgres

docker log postgres returns:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
2020-02-29 18:03:53.948 UTC [35] LOG:  could not link file "pg_wal/xlogtemp.35" to "pg_wal/000000010000000000000001": Operation not permitted
2020-02-29 18:03:53.950 UTC [35] FATAL:  could not open file "pg_wal/000000010000000000000001": No such file or directory
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data"
running bootstrap script ...

I followed the notes from the #28, I can see the local data directory in boot2docker image when I open it through parallels, Docker changes the directory group (or user) to 999. I'm 99% confident this is the same setup I used on another machine last year and it worked fine.

P.S. Or is this related to #45 as the error refers to failing to create a file link? Apparently, this isn't an issue with other drivers.

[romankulikov]

@romankulikov I'm running into what might be the same issue with:

docker run -d --name postgres -e POSTGRES_PASSWORD=postgres -v /Users/ianbytchek/Documents/postgres:/var/lib/postgresql/data -p 5432:5432 postgres

docker log postgres returns:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
2020-02-29 18:03:53.948 UTC [35] LOG:  could not link file "pg_wal/xlogtemp.35" to "pg_wal/000000010000000000000001": Operation not permitted
2020-02-29 18:03:53.950 UTC [35] FATAL:  could not open file "pg_wal/000000010000000000000001": No such file or directory
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data"
running bootstrap script ...

I followed the notes from the #28, I can see the local data directory in boot2docker image when I open it through parallels, Docker changes the directory group (or user) to 999. I'm 99% confident this is the same setup I used on another machine last year and it worked fine.

P.S. Or is this related to #45 as the error refers to failing to create a file link? Apparently, this isn't an issue with other drivers.

The thing is that Parallels Shared Folders don't support creating hard links. That's why you get such error. Here it is: https://github.com/Parallels/docker-machine-parallels/issues/45.

But this could be workarounded by the algorithm described by the topic-starter https://github.com/Parallels/docker-machine-parallels/issues/71#issuecomment-348153883.

[AndreyMashukov]

Solution is

services:
  postgres:
    # ...
    volumes:
      # parallels on mac os does not support hard links, that's why we have to do it
      - postgres-data:/data/postgres
    environment:
      - PGDATA=/data/postgres
    # ...

volumes:
    postgres-data:
        name: postgres_volume_name

[KatoshiPL]

Hi guys I have similar error as @romankulikov. I am trying to use docker-compose(I am new to docker) for my API and postgres database. Her is how my docker compose file looks:

version: '3.9'

services:
    web:
        container_name: sourceryapi
        build: .
        ports:
            - "8000:80"
        depends_on:
            database:
                    condition: service_healthy
    database:
        container_name: postgres
        image: postgres:latest
        ports: 
            - "5433:5432"
        environment:
            - POSTGRES_PASSWORD=admin
        volumes:
            - dbdata:/var/lib/postgresql/data
            #- ./init.sql:/docker-entrypoint-initdb.d/init.sql
        healthcheck:
            test: ["CMD-SHELL", "pg_isready -U postgres"]
            interval: 10s
            timeout: 25s
            retries: 5
volumes:
  dbdata:

And I am getting these output after docker-compose up:

Creating volume "postgres_volume_name" with default driver
Creating postgres ... 

Creating postgres ... done

Creating sourceryapi ... 

Creating sourceryapi ... done

Attaching to postgres, sourceryapi
postgres    | The files belonging to this database system will be owned by user "postgres".
postgres    | This user must also own the server process.
postgres    | 
postgres    | The database cluster will be initialized with locale "en_US.utf8".
postgres    | The default database encoding has accordingly been set to "UTF8".
postgres    | The default text search configuration will be set to "english".
postgres    | 
postgres    | Data page checksums are disabled.
postgres    | 
postgres    | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres    | creating subdirectories ... ok
postgres    | selecting dynamic shared memory implementation ... posix
postgres    | selecting default max_connections ... 100
postgres    | selecting default shared_buffers ... 128MB
postgres    | selecting default time zone ... Etc/UTC
postgres    | creating configuration files ... ok
postgres    **| running bootstrap script ... 2022-04-29 08:25:58.163 UTC [43] LOG:  could not link file "pg_wal/xlogtemp.43" to "pg_wal/000000010000000000000001": Permission denied
postgres    | 2022-04-29 08:25:58.167 UTC [43] FATAL:  could not open file "pg_wal/000000010000000000000001": No such file or directory**
postgres    | child process exited with exit code 1
postgres    | initdb: removing contents of data directory "/var/lib/postgresql/data"
postgres exited with code 4294967295
sourceryapi | Unhandled exception. System.IO.FileNotFoundException: Error reading the C:\app\ directory.

sourceryapi |    at System.IO.FileSystemWatcher.StartRaisingEvents()

sourceryapi |    at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()

sourceryapi |    at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)

sourceryapi |    at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()

sourceryapi |    at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)

sourceryapi |    at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)

sourceryapi |    at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()

sourceryapi |    at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)

sourceryapi |    at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)

sourceryapi |    at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)

sourceryapi |    at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)

sourceryapi |    at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)

sourceryapi |    at Microsoft.Extensions.Configuration.ConfigurationExtensions.Add[TSource](IConfigurationBuilder builder, Action`1 configureSource)

sourceryapi |    at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, IFileProvider provider, String path, Boolean optional, Boolean reloadOnChange)

sourceryapi |    at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, String path, Boolean optional, Boolean reloadOnChange)

sourceryapi |    at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.<>c__DisplayClass11_0.<ConfigureDefaults>b__1(HostBuilderContext hostingContext, IConfigurationBuilder config)

sourceryapi |    at Microsoft.AspNetCore.Hosting.BootstrapHostBuilder.RunDefaultCallbacks(ConfigurationManager configuration, HostBuilder innerBuilder)

sourceryapi |    at Microsoft.AspNetCore.Builder.WebApplicationBuilder..ctor(WebApplicationOptions options, Action`1 configureDefaults)

sourceryapi |    at Microsoft.AspNetCore.Builder.WebApplication.CreateBuilder(String[] args)

sourceryapi |    at Program.<Main>$(String[] args) in C:\source\Program.cs:line 8
sourceryapi exited with code 3762504530


Seriously it is frustrating, I cannot find any solution to this...maybe you can help?