dokku / dokku-postgres

a postgres plugin for dokku
MIT License
470 stars 96 forks source link

Read-only file system on postgres:create #292

Closed AnisSjd closed 1 year ago

AnisSjd commented 1 year ago

Description of problem

I was following the official app deployment tuto at https://dokku.com/docs/deployment/application-deployment/ with my dokku image (from https://hub.docker.com/r/dokku/dokku), but get stuck with:

$ docker compose exec dokku dokku postgres:create railsdatabase
Error response from daemon: error while creating mount source path '/var/lib/dokku/services/postgres/railsdatabase/data': mkdir /var/lib/dokku: read-only file system
Error: failed to start containers: eedca8efbc4bde92824a379e4455a2a70fad8caec4c0e11490cc11ea9638d084

How reproducible

It happens every time on my machine, even when I clean the containers created by dokku.

Steps to Reproduce

echo "postgres: https://github.com/dokku/dokku-postgres.git" > plugin-list;

docker pull dokku/dokku:0.30.2;

docker run -d \
  --name dokku-test \
  --volume ${PWD}/plugin-list:/mnt/dokku/plugin-list \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  dokku/dokku:0.30.2 &

# sleep 180 # wait Runit to be up

docker exec dokku-test dokku apps:create ruby-getting-started

docker exec dokku-test dokku postgres:create railsdatabase

Actual Results

Error response from daemon: error while creating mount source path '/var/lib/dokku/services/postgres/railsdatabase/data': mkdir /var/lib/dokku: read-only file system
Error: failed to start containers: eedca8efbc4bde92824a379e4455a2a70fad8caec4c0e11490cc11ea9638d084

Expected Results

No error + DB service is created so I can continue the tuto

Environment Information

dokku report output

-----> uname: Linux 0442979ce7de 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
-----> memory: 
                     total        used        free      shared  buff/cache   available
       Mem:           7759         314        3772           1        3672        7149
       Swap:             0           0           0
-----> docker version: 
       Client: Docker Engine - Community
        Version:           23.0.1
        API version:       1.41 (downgraded from 1.42)
        Go version:        go1.19.5
        Git commit:        a5ee5b1
        Built:             Thu Feb  9 19:46:56 2023
        OS/Arch:           linux/amd64
        Context:           default

       Server:
        Engine:
         Version:          20.10.17
         API version:      1.41 (minimum version 1.12)
         Go version:       go1.18.10
         Git commit:       a89b842
         Built:            Tue Mar  7 06:30:14 2023
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          v1.6.6
         GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
        runc:
         Version:          1.1.2
         GitCommit:        
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client:
        Context:    default
        Debug Mode: true
        Plugins:
         compose: Docker Compose (Docker Inc.)
           Version:  v2.16.0
           Path:     /usr/libexec/docker/cli-plugins/docker-compose

       Server:
        Containers: 4
         Running: 1
         Paused: 0
         Stopped: 3
        Images: 10
        Server Version: 20.10.17
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
        runc version: 
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: default
         cgroupns
        Kernel Version: 5.15.0-69-generic
        Operating System: Ubuntu Core 18
        OSType: linux
        Architecture: x86_64
        CPUs: 4
        Total Memory: 7.578GiB
        Name: vps-9a76c2a9
        ID: X7CQ:6YVP:FU5N:QEZS:C7YE:E5H7:J4NQ:63GM:C7LC:DIBZ:PILJ:5TXB
        Docker Root Dir: /var/snap/docker/common/var-lib-docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false

-----> git version: git version 2.25.1
-----> sigil version: 0.9.0build+bc921b7
-----> herokuish version: 
Unable to find image 'gliderlabs/herokuish:latest-20' locally
latest-20: Pulling from gliderlabs/herokuish
06d39c85623a: Pulling fs layer
1aebbe1b55dc: Pulling fs layer
cbb9b6b01f72: Pulling fs layer
3d5ce0cd749c: Pulling fs layer
3d5ce0cd749c: Waiting
98cab72ffa42: Pulling fs layer
a786c984824a: Pulling fs layer
98cab72ffa42: Waiting
d66737854f1e: Pulling fs layer
6dba6e891c6b: Pulling fs layer
132bb2747060: Pulling fs layer
70f4c885c73c: Pulling fs layer
d66737854f1e: Waiting
6dba6e891c6b: Waiting
132bb2747060: Waiting
70f4c885c73c: Waiting
1aebbe1b55dc: Download complete
cbb9b6b01f72: Verifying Checksum
cbb9b6b01f72: Download complete
06d39c85623a: Verifying Checksum
06d39c85623a: Download complete
3d5ce0cd749c: Verifying Checksum
3d5ce0cd749c: Download complete
a786c984824a: Verifying Checksum
a786c984824a: Download complete
d66737854f1e: Verifying Checksum
d66737854f1e: Download complete
132bb2747060: Verifying Checksum
132bb2747060: Download complete
6dba6e891c6b: Verifying Checksum
6dba6e891c6b: Download complete
70f4c885c73c: Verifying Checksum
70f4c885c73c: Download complete
06d39c85623a: Pull complete
1aebbe1b55dc: Pull complete
cbb9b6b01f72: Pull complete
3d5ce0cd749c: Pull complete
98cab72ffa42: Verifying Checksum
98cab72ffa42: Download complete
98cab72ffa42: Pull complete
a786c984824a: Pull complete
d66737854f1e: Pull complete
6dba6e891c6b: Pull complete
132bb2747060: Pull complete
70f4c885c73c: Pull complete
Digest: sha256:2f46394d60dbf5839339b7f592596fbaed377201cfe63d37e434689424de6dfb
Status: Downloaded newer image for gliderlabs/herokuish:latest-20
       herokuish: v0.5.41
       buildpacks:
         heroku-buildpack-multi     v1.2.0
         heroku-buildpack-ruby      v252
         heroku-buildpack-nodejs    v203
         heroku-buildpack-clojure   v90
         heroku-buildpack-python    v226
         heroku-buildpack-java      v72
         heroku-buildpack-gradle    v39
         heroku-buildpack-scala     v96
         heroku-buildpack-play      v26
         heroku-buildpack-php       v231
         heroku-buildpack-go        v171
         heroku-buildpack-nginx     v22
         buildpack-null             v3
-----> dokku version: dokku version 0.30.2
-----> plugn version: plugn: 0.12.0build+3a27594
-----> dokku plugins: 
         00_dokku-standard    0.30.2 enabled    dokku core standard plugin
         20_events            0.30.2 enabled    dokku core events logging plugin
         app-json             0.30.2 enabled    dokku core app-json plugin
         apps                 0.30.2 enabled    dokku core apps plugin
         builder              0.30.2 enabled    dokku core builder plugin
         builder-dockerfile   0.30.2 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.30.2 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.30.2 enabled    dokku core builder-lambda plugin
         builder-null         0.30.2 enabled    dokku core builder-null plugin
         builder-pack         0.30.2 enabled    dokku core builder-pack plugin
         buildpacks           0.30.2 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.30.2 enabled    dokku core caddy-vhosts plugin
         certs                0.30.2 enabled    dokku core certificate management plugin
         checks               0.30.2 enabled    dokku core checks plugin
         common               0.30.2 enabled    dokku core common plugin
         config               0.30.2 enabled    dokku core config plugin
         cron                 0.30.2 enabled    dokku core cron plugin
         docker-options       0.30.2 enabled    dokku core docker-options plugin
         domains              0.30.2 enabled    dokku core domains plugin
         enter                0.30.2 enabled    dokku core enter plugin
         git                  0.30.2 enabled    dokku core git plugin
         haproxy-vhosts       0.30.2 enabled    dokku core haproxy-vhosts plugin
         logs                 0.30.2 enabled    dokku core logs plugin
         network              0.30.2 enabled    dokku core network plugin
         nginx-vhosts         0.30.2 enabled    dokku core nginx-vhosts plugin
         plugin               0.30.2 enabled    dokku core plugin plugin
         postgres             1.33.0 enabled    dokku postgres service plugin
         proxy                0.30.2 enabled    dokku core proxy plugin
         ps                   0.30.2 enabled    dokku core ps plugin
         registry             0.30.2 enabled    dokku core registry plugin
         repo                 0.30.2 enabled    dokku core repo plugin
         resource             0.30.2 enabled    dokku core resource plugin
         run                  0.30.2 enabled    dokku core run plugin
         scheduler            0.30.2 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.30.2 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.30.2 enabled    dokku core scheduler-null plugin
         shell                0.30.2 enabled    dokku core shell plugin
         ssh-keys             0.30.2 enabled    dokku core ssh-keys plugin
         storage              0.30.2 enabled    dokku core storage plugin
         trace                0.30.2 enabled    dokku core trace plugin
         traefik-vhosts       0.30.2 enabled    dokku core traefik-vhosts plugin

How (deb/make) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

Where: macOS Ventura 13.3 @ Macbook Air M1 How: Docker image at https://hub.docker.com/r/dokku/dokku

Additional information

josegonzalez commented 1 year ago

What host os are you running Dokku on?

AnisSjd commented 1 year ago

Host: macOS Ventura 13.3 Docker image: Linux 0442979ce7de 5.15.0-69-generic (from https://hub.docker.com/r/dokku/dokku)

josegonzalez commented 1 year ago

The docker installation notes show environment variables and mounts that need to be set. The host directories should be writeable - if not, you'll get this error. I think thats expected on OSX, so you'll need to use a host path other than /var/lib/dokku for data.

AnisSjd commented 1 year ago

You are right, I used the instructions at https://hub.docker.com/r/dokku/dokku, instead of the (more complete?) ones at https://dokku.com/docs/getting-started/install/docker/ .

I fixed my previous command, but the issue remains.

Note that I intentionally haven't mounted my host /var/lib/dokku to avoid bind mount issues as the one you mentionned.

Here is the updated command:

docker run --rm \
  --env DOKKU_HOSTNAME=dokku.me \
  --env DOKKU_HOST_ROOT=/var/lib/dokku/home/dokku \
  --env DOKKU_LIB_HOST_ROOT=/var/lib/dokku/var/lib/dokku \
  --name dokku-test \
  --publish 3022:22 \
  --publish 8080:80 \
  --publish 8443:443 \
  --volume ${PWD}/plugin-list:/mnt/dokku/plugin-list \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  dokku/dokku:0.30.2 &
josegonzalez commented 1 year ago

The datastore plugins store data on disk and use the mounted docker socket, so all the directories are mounted from the host.

The host root env vars are used to specify where to mount directories from. Since you set them relative to /var/lib/dokku, that's where data gets mounted from.

AnisSjd commented 1 year ago

OK, the following params let me create the database:

docker run --rm \
  --env DOKKU_HOSTNAME=dokku.me \
  --env DOKKU_HOST_ROOT=${PWD}/home/dokku \
  --env DOKKU_LIB_HOST_ROOT=${PWD}/var/lib/dokku \
  --name dokku-test \
  --publish 3022:22 \
  --publish 8080:80 \
  --publish 8443:443 \
  --volume ${PWD}/var/lib/dokku:/mnt/dokku \
  --volume ${PWD}/plugin-list:/mnt/dokku/plugin-list \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  dokku/dokku:0.30.2 &

I understand now the importance of DOKKU_LIB_HOST_ROOT; however I'm still unsure of its exact use and limitations, since I can't find enough documentation about it, both at https://hub.docker.com/r/dokku/dokku and https://dokku.com/docs/getting-started/install/docker/.

Thanks for help.

jbosse commented 1 month ago

@josegonzalez I am running into the same issue. However I am running on Ubuntu server, from the bootstrap.sh. My docker report is here

sudo dokku postgres:create sample_db
Error response from daemon: error while creating mount source path '/var/lib/dokku/services/postgres/sample_db/data': mkdir /var/lib/dokku: read-only file system
Error: failed to start containers: 835486d8356fff70d64cf8ca1da254db161f9567212d5766643869f6060fdfd3
jbosse commented 1 month ago

I managed to resolve the issue by uninstalling the snap docker (snap remove docker) and reinstalling docker following https://docs.docker.com/engine/install/ubuntu/ then rebooting.