Files in Synchronized file share are not being deleted

Description

It's not possible to remove some files created within container while using Synchronized file shares.

When files are being deleted on the host, they disappear for a moment and are recreated soon after. I've looked at the files inside container while deleting them on the host and they seem not to disappear. The only way to remove those files is to stop the container and remove Synchronized file share.

Docker reports problems with those files, but nothing else can be done.

Screenshot 2024-04-16 at 13 51 16

Reproduce

Files: docker-compose.yml

services:
  test:
    container_name: test
    build: .
    volumes:
      - ./app:/app
    command: tail -f /dev/null

Dockerfile

FROM ubuntu:jammy

RUN apt update \
    && apt install -y \
    ruby-full \
    build-essential

WORKDIR /app

# Install bundler
RUN gem install bundler -v '2.2.22'

# Install gems in app directory
ENV BUNDLE_PATH /app/.bundle
ENV BUNDLE_BIN $BUNDLE_PATH/bin

app/Gemfile

source 'https://rubygems.org'
ruby '3.0.2'
gem 'cucumber-rails', '2.2.0'

Create Synchronized file share pointing to app folder
Run docker-compose up -d
Run docker exec -it test bash and inside container run bundle install (this will create Gemfile.lock and .bundle folder) - Docker will report problems with sync share at that stage
Try to remove .bundle folder on the host (in finder or IDE) - it disappears but is recreated by docker.

Expected behavior

Docker does not report any problems with files when using Synchronized file shares
Files in Synchronized file share are removed permanently when deleted on host

docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:14:46 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:02 2024
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    26.0.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     /Users/konradkucharski/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     /Users/konradkucharski/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     /Users/konradkucharski/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/konradkucharski/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /Users/konradkucharski/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/konradkucharski/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /Users/konradkucharski/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/konradkucharski/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     /Users/konradkucharski/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/konradkucharski/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/konradkucharski/.docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 5
  Running: 4
  Paused: 0
  Stopped: 1
 Images: 59
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.22-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 9.708GiB
 Name: docker-desktop
 ID: 3965164e-7cc3-4313-aaef-1083363097ad
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/konradkucharski/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

CF1919CE-7AC4-4466-8329-ADF65B1381F4/20240416120155

Additional Info

No response

Hey @konrad-o, thanks for the very detailed report and reproducer - it made the issue very easy to reproduce and debug.

The crux of the issue is that one of the packages installed by Bundler includes directories (in .bundle/ruby/3.0.0/gems/cucumber-rails-2.2.0/tmp/aruba/test_app/tmp/cache/assets/sprockets/v3.0/) whose names differ only in case (e.g. MZ vs mz). This is fine on the Linux ext4 filesystem (which is case-preserving and case-sensitive), but it doesn't work on APFS and NTFS (which are case-preserving but case-insensitive). These case conflicts manifest as the "file exists" errors.

When installing natively on macOS, bundle doesn't care about case conflicts and seems to just merge the contents of the folders, but Synchronized File Shares are a little more cautious and won't allow case conflicts to overwrite each other on the macOS side, so these conflicts are held in a "pending" state (sort of — the details are a little more complex, but conceptually similar), meaning that when the case-conflicting files are deleted, Synchronized File Shares then propagates the pending conflicting files through, making their parent directory reappear after deletion. This is the "by design" behavior, but we can certainly add some heuristic tweaks for Docker Desktop to improve the experience. I've added a few tickets internally to clarify that the "file exists" messages indicate a case conflict and to make the deletion behavior for case conflicts a little more user-friendly.

Unfortunately neither of these fixes will address the core issue here, which is that Linux and macOS (+Windows) have fundamentally different case sensitivity/preservation behavior.

As a workaround, I would suggest adding a .syncignore file containing .bundle/**/tmp/aruba/test_app/tmp/cache/assets/sprockets/v3.0 to app to exclude this directory.

The actual ignore rule could be more broad or more specific, depending on the files that you want to include/exclude in synchronization back to the host. I'm happy to provide more input on the exact ignore strategy if that's helpful. Note that you'll probably want to delete and recreate the Synchronized File Share after adding the .syncignore file.

Hey @xenoscopic! Thanks for the quick reply. Your answer pointed me into the right direction. So the actual issue was that bundle install was creating files with the same folder paths, but with different casing, e.g.:

path/Ab/file1
path/ab/file2

Without synchronized file shares in case-insensitive file system, both files were created in the same folder (whichever came first), so it was like

path/Ab/file1
path/Ab/file2

both on host as inside running container. Docker wasn't complaining about it (not like in synchronized file shares), so it went unnoticed. I think that your workaround would not work for us. As far as I understand how .syncignore works it'd mean that those files would exist only on host and container needs them while running. Also the point of placing gem (package) files on host is to have them persistent across development process (starting/stopping of containers) so we'd have to either run bundle install on every container startup or run it during image build - both approaches are inconvenient. Thing that worked for us it to create a new APFS (Case-sensitive) volume and run our project there.

But I still would say that not to being able to remove files in above situation, is a bug in Synchronized file shares. What do you think? BTW - is it possible to see whole list of files that cause problems? If there's many, it shows only around 10 of them.

Hey again! Just to clarify regarding the .syncignore: it would actually keep those files in the container but not on the host (not the other way around). The ignores apply symmetrically, so anything targeted by .syncignore won't be sync'd from host->container or from container->host. This allows you to have files that exist on the host but not in the container, but also the other way around (which I think might solve this case).

Depending on whether or not you need the files in that conflicting directory, you could still rely on persistence. So if you set up the .syncignore to exclude only that directory, then install inside the container, and sync back to the host, then the Synchronized File Share would still have all the files except that particular test fixture. I'm not sure if that would be sufficient for your needs.

Creating a case-sensitive APFS volume is also a great idea.

Regarding the removal of the files, I would classify it as a "UX bug". The files are actually being deleted, but what's happening is that the "pending" case-conflicted files (which are waiting to be sync'd) are being brought through after the deletion (which necessitates creating their parent directories, which is allowed by Synchronized File Shares as long as there's nothing blocking it). I've created a ticket to add a heuristic to handle this behavior so that if all of the "pending" files are due to case conflicts, that we just allow them to be deleted in a way that feels more intuitive.

At the moment there's not a way to view the full list on problems and conflicts, but that's a reasonable ask. I'll create an additional ticket for exploring that idea.

Thanks for .syncignore clarification. All works now as expected :)

docker / for-mac