VirtioFS changes in 4.34 break initial startup of PostgreSQL container

robmoore-i commented 2 months ago

Description

We're on MacOS 14.6.1 using Docker Desktop 4.34.0 and we're using the image postgres:16.4-bookworm.

We mount an empty directory onto /mnt/data and set PGDATA=/mnt/data/postgresql. If it doesn't already exist, we create PGDATA on container startup (using mkdir), before invoking the default entrypoint with exec /usr/local/bin/docker-entrypoint.sh postgres. To remove vulnerabilities from the image, in our Dockerfile we run rm /usr/local/bin/gosu as part of a general purge of unused code in the container, and we also set USER postgres. This code has been working unchanged since March on older versions of Docker Desktop which use VirtioFS. Our container definition also works fine on other platforms, including e.g. OpenShift.

Upon upgrading to 4.34.0 a teammate reported that some tests were failing locally on their Mac because Postgres was no longer able to start up, due to a permissions issue. While debugging the problem, I added the following logs to the container entrypoint script:

# echo "Running as user '$(whoami)' ($(id))"
Running as user 'postgres' (uid=999(postgres) gid=999(postgres) groups=999(postgres),101(ssl-cert))

# stat -c "name=(%n) file_type=(%F) owner_group=(%g/%G) owner_user=(%u/%U) permission_bits=(%A) mount_point=(%m)" "$PGDATA"
name=(/mnt/data/postgresql) file_type=(directory) owner_group=(999/postgres) owner_user=(999/postgres) permission_bits=(drwx------) mount_point=(/mnt/data)

I think this shows that the user starting the postgres server is the owner of the data directory.

And yet when the database starts, it immediately crashes:

FATAL:  data directory "/mnt/data/postgresql" has wrong ownership
HINT:  The server must be started by the user that owns the data directory

Changing the file sharing implementation in Docker Desktop from VirtioFS to gRPC FUSE fixes the problem immediately but is a bad solution because it requires everyone in the team to know about the problem and know how to fix it. I also unsuccessfully tried to use chmod -R and chown -R to make sure that the created directory has the right permissions, but this had no effect, and why would it? - The stat call above suggests that the directory is owned by the postgres user with uid 999. I don't know that there's anything I can do within the container to fix this problem.

This seems like a bug in Docker to me, but let me know if you think there's something else in our image definition I should try. Let me know if I can help further, thanks.

This may look similar to https://github.com/docker/for-mac/issues/6270 but this affects a much more recent version and is a different problem.

Reproduce

As described above.

Expected behavior

The PostgreSQL container should start up without issue, as it did in 4.33.0.

docker version

Client:
 Version:           27.2.0
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Aug 27 14:14:45 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.34.0 (165256)
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:15:41 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.2.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     /Users/rmoore/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     /Users/rmoore/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/rmoore/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/rmoore/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/rmoore/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/rmoore/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/rmoore/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/rmoore/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/rmoore/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     /Users/rmoore/.docker/cli-plugins/docker-scout

Server:
 Containers: 37
  Running: 0
  Paused: 0
  Stopped: 37
 Images: 363
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.4-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 62.69GiB
 Name: docker-desktop
 ID: 6526bca5-9abd-48c2-87eb-9de37472d30f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/rmoore/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  192.168.0.0/16
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

Uploading of company data is probably not allowed.

Additional Info

No response

abuchanan-airbyte commented 2 months ago

I've run into this issue as well, and I confirmed that downgrading to Docker Desktop 4.33.0 (160616) (with "ServerVersion": "27.1.1") resolved the issue. I upgraded and downgraded a couple times to be sure.

Note, the description overlaps with https://github.com/docker/for-mac/issues/7406 somewhat.