Cannot set data_directory with new postgres service

[danjac]

Description of problem

I have an existing copy PostgreSQL database directory under a volume /mnt/volume-xxx. The volume is mounted and is accessible. The data dir is /mnt/volume-xxx/postgres/data. The data was created using PostgreSQL 14.8, so the service image should use the same version.

I wish to create a new Dokku postgres service and set the data_directory to point to this directory (I would prefer to just create the new service and set data_directory in postgresql.conf, but this appears to be ignored on restarting the service):

dokku postgres:create my-postgres-service --image-version "14.8" --config-options "--data_directory=/mnt/volume-xxx/postgres/data

I then get a number of errors:

       Waiting for container to be ready
Waiting for dokku-postgres-my-postgres-service:5432  ..............................  ERROR: unable to connect
=====> Start of my-postgres-service container output
       The files belonging to this database system will be owned by user "postgres".
       This user must also own the server process.
       The database cluster will be initialized with locale "en_US.utf8".
       The default database encoding has accordingly been set to "UTF8".
       The default text search configuration will be set to "english".
       Data page checksums are disabled.
       fixing permissions on existing directory /var/lib/postgresql/data ... ok
       creating subdirectories ... ok
       selecting dynamic shared memory implementation ... posix
       selecting default max_connections ... 100
       selecting default shared_buffers ... 128MB
       selecting default time zone ... Etc/UTC
       creating configuration files ... ok
       running bootstrap script ... ok
       performing post-bootstrap initialization ... ok
       syncing data to disk ... ok
       Success. You can now start the database server using:
           pg_ctl -D /var/lib/postgresql/data -l logfile start
       initdb: warning: enabling "trust" authentication for local connections
       You can change this by editing pg_hba.conf or using the option -A, or
       --auth-local and --auth-host, the next time you run initdb.
       2023-07-08 07:45:50.363 GMT [44] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       waiting for server to start....2023-07-08 07:45:50.381 GMT [49] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       2023-07-08 07:45:50.381 UTC [49] FATAL:  data directory "/mnt/volume-xxx/postgres/data" does not exist
       pg_ctl: could not start server
       Examine the log output.
        stopped waiting
       PostgreSQL Database directory appears to contain a database; Skipping initialization
       2023-07-08 07:45:50.827 GMT [1] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       2023-07-08 07:45:50.827 UTC [1] FATAL:  data directory "/mnt/volume-xxx/postgres/data" does not exist
       PostgreSQL Database directory appears to contain a database; Skipping initialization
       2023-07-08 07:45:51.270 GMT [1] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       2023-07-08 07:45:51.271 UTC [1] FATAL:  data directory "/mnt/volume-xxx/postgres/data" does not exist
       PostgreSQL Database directory appears to contain a database; Skipping initialization
       2023-07-08 07:45:51.906 GMT [1] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       2023-07-08 07:45:51.907 UTC [1] FATAL:  data directory "/mnt/volume-xxx/postgres/data" does not exist
       PostgreSQL Database directory appears to contain a database; Skipping initialization
       2023-07-08 07:45:52.966 GMT [1] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       2023-07-08 07:45:52.966 UTC [1] FATAL:  data directory "/mnt/volume-xxx/postgres/data" does not exist
       PostgreSQL Database directory appears to contain a database; Skipping initialization
       2023-07-08 07:45:54.819 GMT [1] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       2023-07-08 07:45:54.820 UTC [1] FATAL:  data directory "/mnt/volume-xxx/postgres/data" does not exist
       PostgreSQL Database directory appears to contain a database; Skipping initialization
       2023-07-08 07:45:58.270 GMT [1] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       2023-07-08 07:45:58.270 UTC [1] FATAL:  data directory "/mnt/volume-xxx/postgres/data" does not exist
       PostgreSQL Database directory appears to contain a database; Skipping initialization
       2023-07-08 07:46:04.917 GMT [1] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       2023-07-08 07:46:04.917 UTC [1] FATAL:  data directory "/mnt/volume-xxx/postgres/data" does not exist
       PostgreSQL Database directory appears to contain a database; Skipping initialization
       2023-07-08 07:46:17.971 GMT [1] LOG:  skipping missing configuration file "/mnt/volume-xxx/postgres/data/postgresql.auto.conf"
       2023-07-08 07:46:17.971 UTC [1] FATAL:  data directory "/mnt/volume-xxx/postgres/data" does not exist
=====> End of my-postgres-service container output

Not only does the data directory exist, but it has the correct ownership ("dokku:dokku").

Environment Information

dokku report APP_NAME output

-----> uname: Linux ubuntu-4gb-hel1-1 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
-----> memory: 
                      total        used        free      shared  buff/cache   available
       Mem:            7747         342        2854           3        4550        7091
       Swap:           1023          57         966
-----> docker version: 
       Client: Docker Engine - Community
        Version:           24.0.3
        API version:       1.43
        Go version:        go1.20.5
        Git commit:        3713ee1
        Built:             Wed Jul  5 20:44:47 2023
        OS/Arch:           linux/amd64
        Context:           default

       Server: Docker Engine - Community
        Engine:
         Version:          24.0.3
         API version:      1.43 (minimum version 1.12)
         Go version:       go1.20.5
         Git commit:       1d9c861
         Built:            Wed Jul  5 20:44:47 2023
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.6.21
         GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
        runc:
         Version:          1.1.7
         GitCommit:        v1.1.7-0-g860f061
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client: Docker Engine - Community
        Version:    24.0.3
        Context:    default
        Debug Mode: true
        Plugins:
         buildx: Docker Buildx (Docker Inc.)
           Version:  v0.11.1
           Path:     /usr/libexec/docker/cli-plugins/docker-buildx
         compose: Docker Compose (Docker Inc.)
           Version:  v2.19.1
           Path:     /usr/libexec/docker/cli-plugins/docker-compose
         scan: Docker Scan (Docker Inc.)
           Version:  v0.23.0
           Path:     /usr/libexec/docker/cli-plugins/docker-scan

       Server:
        Containers: 4
         Running: 2
         Paused: 0
         Stopped: 2
        Images: 8
        Server Version: 24.0.3
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Using metacopy: false
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
        runc version: v1.1.7-0-g860f061
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: builtin
         cgroupns
        Kernel Version: 5.15.0-76-generic
        Operating System: Ubuntu 22.04.2 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 4
        Total Memory: 7.566GiB
        Name: ubuntu-4gb-hel1-1
        ID: I66S:3YBF:4347:6RK7:DEAO:XCA5:VU7U:4BO5:EEMB:RXOW:DKUF:SXMK
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
         File Descriptors: 38
         Goroutines: 48
         System Time: 2023-07-08T08:02:02.206773099Z
         EventsListeners: 2
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false

-----> git version: git version 2.34.1
-----> sigil version: 0.9.0build+bc921b7
-----> herokuish version: 
       herokuish: v0.6.0
       buildpacks:
         heroku-buildpack-multi     v1.2.0
         heroku-buildpack-ruby      v254
         heroku-buildpack-nodejs    v213
         heroku-buildpack-clojure   v90
         heroku-buildpack-python    v232
         heroku-buildpack-java      v72
         heroku-buildpack-gradle    v39
         heroku-buildpack-scala     v96
         heroku-buildpack-play      v26
         heroku-buildpack-php       v234
         heroku-buildpack-go        v174
         heroku-buildpack-nginx     v23
         buildpack-null             v3
-----> dokku version: dokku version 0.29.3
-----> plugn version: plugn: 0.12.0build+3a27594
-----> dokku plugins: 
         00_dokku-standard    0.29.3 enabled    dokku core standard plugin
         20_events            0.29.3 enabled    dokku core events logging plugin
         app-json             0.29.3 enabled    dokku core app-json plugin
         apps                 0.29.3 enabled    dokku core apps plugin
         builder              0.29.3 enabled    dokku core builder plugin
         builder-dockerfile   0.29.3 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.29.3 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.29.3 enabled    dokku core builder-lambda plugin
         builder-null         0.29.3 enabled    dokku core builder-null plugin
         builder-pack         0.29.3 enabled    dokku core builder-pack plugin
         buildpacks           0.29.3 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.29.3 enabled    dokku core caddy-vhosts plugin
         certs                0.29.3 enabled    dokku core certificate management plugin
         checks               0.29.3 enabled    dokku core checks plugin
         common               0.29.3 enabled    dokku core common plugin
         config               0.29.3 enabled    dokku core config plugin
         cron                 0.29.3 enabled    dokku core cron plugin
         docker-options       0.29.3 enabled    dokku core docker-options plugin
         domains              0.29.3 enabled    dokku core domains plugin
         enter                0.29.3 enabled    dokku core enter plugin
         git                  0.29.3 enabled    dokku core git plugin
         letsencrypt          0.20.1 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.29.3 enabled    dokku core logs plugin
         network              0.29.3 enabled    dokku core network plugin
         nginx-vhosts         0.29.3 enabled    dokku core nginx-vhosts plugin
         plugin               0.29.3 enabled    dokku core plugin plugin
         postgres             1.34.0 enabled    dokku postgres service plugin
         proxy                0.29.3 enabled    dokku core proxy plugin
         ps                   0.29.3 enabled    dokku core ps plugin
         redirect             0.9.0 enabled    Plugin for managing application redirects
         redis                1.34.0 enabled    dokku redis service plugin
         registry             0.29.3 enabled    dokku core registry plugin
         repo                 0.29.3 enabled    dokku core repo plugin
         resource             0.29.3 enabled    dokku core resource plugin
         run                  0.29.3 enabled    dokku core run plugin
         scheduler            0.29.3 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.29.3 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.29.3 enabled    dokku core scheduler-null plugin
         shell                0.29.3 enabled    dokku core shell plugin
         ssh-keys             0.29.3 enabled    dokku core ssh-keys plugin
         storage              0.29.3 enabled    dokku core storage plugin
         trace                0.29.3 enabled    dokku core trace plugin
         traefik-vhosts       0.29.3 enabled    dokku core traefik-vhosts plugin
 !     App my-postgres-service does not exist

This is required! Issues missing this information may be closed.

For problems affecting all applications, the report output for a broken application is useful for our debugging. In these cases, you may run dokku report without any arguments to display the top-level reporting information.

How (deb/make) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

Hetzner Cloud Ubuntu 22.04.2 LTS x86_64

[josegonzalez]

Your host data directories are not mounted into containers, and we don't support custom paths for managed services. If you want to do this:

• create the service
• stop the service
• symlink the data directory into place (it should be in the output of dokku postgres:info $SERVICE)
• start the service

the above is untested but should work. Please back up your existing data before continuing.

All that said, I don't think it's a great idea for us to add functionality allowing folks to customize the data directory otherwise, simply because of complexity of this sort of feature.

Hope that helps.

[danjac]

Symlinking worked, thanks. This might be a common requirement when needing to move data PostgreSQL in an existing application due to disk space issues, so would be good to document (as the standard approach of setting data_directory won't work).

[josegonzalez]

Pull requests welcome.