Missing file system events on mounts

[velikovelikov]

Expected behavior

Run container with shared directory using -- mount flag:

docker run -it --rm --mount 'type=bind,source=LOCAL_DIR,target=/CONTAINER_DIR' ubuntu:17.04 /bin/bash

File system events must be fired in the container

Actual behavior

File system events are missing

Information

Running container with shared directory from host has different behavior depending on the options you are using. File system events are fired on the container if the directory is shared with -v option, and they are missing if we use --mount. As it is written in your documentation their behavior must be similar.

Software versions: Mac OS X: Sierra 10.12.6

users-MacBook-Pro:~ veliko$ docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: ovor9ul30b0bocihujrazh2oc
 Is Manager: true
 ClusterID: 1pagm60q6zg1yhmp3ow8888mc
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 192.168.65.2
 Manager Addresses:
  192.168.65.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: moby
ID: NRLZ:7ZDM:FVEU:ZSSY:C5KA:HKK7:ZWJG:KIUK:JX42:WJKV:UYKD:HEHF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 44
 Goroutines: 154
 System Time: 2017-11-10T09:20:41.146739235Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Docker Diagnose Output:

Docker for Mac: version: 17.09.0-ce-mac35 (69202b202f497d4b6e627c3370781b9e4b51ec78)
macOS: version 10.12.6 (build: 16G29)
logs: /tmp/EE97E13F-1273-4375-8E4C-71B9695A0C51/20171110-111832.tar.gz
[OK]     db.git
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

Steps to reproduce the behavior

1. docker run -it --rm --mount 'type=bind,source=LOCAL_DIR,target=/CONTAINER_DIR' ubuntu:17.04 /bin/bash
2. install inotify-tools using apt-get
3. run inotifywait with some file
4. change watched file from local machine or container
5. No events are tracked

If you run the container with this command: docker run -it --rm -v ~/LOCAL_DIR:/CONTAINER_DIR ubuntu:17.04 /bin/bash

and repeat the same steps events are tracked.

[phyllisstein]

This is still an issue with Docker for Mac Edge, though I can't reproduce it with Docker for Mac Stable. I've noticed that it affects containers run from a Compose file as well. This syntax...

    volumes:
      - .:/usr/src/code
      - modules:/usr/src/code/node_modules

...causes filesystem events to be propagated, whereas the more verbose syntax...

    volumes:
      - source: .
        target: /usr/src/code
        type: bind
      - source: modules
        target: /usr/src/code/node_modules
        type: volume

...does not pass them through to the container.

[thaJeztah]

ping @djs55

[illarionvk]

I have the same issue on Docker for Mac 17.12.0-ce / macOS Sierra 10.12.6.

Filesystem events start propagating if I amend the Compose file as @phyllisstein suggested.

[pkyeck]

I'm using 17.09.1-ce-mac42 (21090) / macOS Sierra 10.12.6. inotify events stopped working during the last couple of days ... but I don't know when I updated or what version I ran before the update.

I'm using docker-compose and declare my volumes like this:

volumes:
      - "./api:/opt/app"

but no events are fired inside the container anymore 😦

[phyllisstein]

Still persists under v18.01.0-ce-mac48 (22004). Using the terse syntax in a Compose file, I get:

root@5bc4164e8016:/usr/src/rupertsberg# inotifywait -rme modify,attrib,move,close_write,create,delete,delete_self *.txt
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
test.txt ATTRIB
test.txt MODIFY

...on changes to test.txt, but using the verbose syntax, inotifywait does not report changes.

[giladv]

any news about this? can't get nodemon to work with the docker api. it does work with docker compose

[Nemo64]

The issue is a little different for me. It seems that file system events are working for a few hours after i restart the docker deamon. But if I let it run though a standby etc, then it'll fail.

[djs55]

Sorry for the delay on this. The original implementation of filesystem events didn't support the --mount option (only the older -v). I've created a prototype fix and am testing it now. I'll keep you all informed of my progress.

Thanks for your reports!

[YRM64]

Djss55,

Please hit me with an update on this issue as soon as one becomes available. Thanks!

[djs55]

I have an experimental development build which should have a fix for this: https://download-stage.docker.com/mac/bysha1/7fc56519ef8858a58fd0eb61d10c958688ff8767/Docker.dmg

Let me know if you get a chance to try it!

[tannerdsilva]

The build appears to be working for me. Thanks @djs55

[YRM64]

Prefixing the following statement as a 'general statement'. It is not meant to be a diagnostic report of the current use case. This is simply an attempt to prevent missing file system events, if at all possible.

Developers can minimize the possibility of losing missing files through rule-making. You can also discover CPUs and CPU cores, SNMP OIDs, network interfaces, ODBC SQL Queries, and be able to discover Window services, all through the rule-making process. You can create a 'discovery rule' (an item that discovers the necessary entities and prototypes) by going to configuration > templates > discovery column. Fill out the form by providing the requested information.

Using the operating system Linux as an example, you can give definition to your own type of discovery(ies) following a certain protocol (Json). You can also set your own parameters, such as Type (the type of check you want performed), Key, (which is built into the agent on many platforms), Update (describing how often you want the agent to search for the discovery), Custom Intervals (this is where you have the opportunity to create the rules of the game). You have a couple of choices here to be as flexible or conservative as you desire, and create a polling schedule. You also have the opportunity to Keep lost resources for a certain period of time/days. You can also provide a description in the discovery column. This column describes what it is you wish to discover, i.e., "Discovery of file systems of different types as defined in global regular expression", or "File System on mounts" for discovery, or File Systems for discovery. You then click on enable, and then Add.

Discovery protection is a way to automatically put into place a number of safeguards that prevent developers from losing files, or file systems. It's a way to establish triggers, create items needed for security and protection, and a savvy way to provide graphical information for networks or various entities on your operating system.

As always, Happy Coding!

[vojkny]

@djs55 when is this going to be released? is this some concrete version?

[djs55]

@knyttl the fix should be in the edge 18.05.0-ce-mac66 and stable 18.06.0-ce-mac70 versions. I'll close this issue but please do let me know if you have further problems.

[paveljanda]

Hi, I am still observing reported issues. Build: Version 18.06.1-ce-mac73 (26764).

[BrianBelhumeur]

I am also seeing the same issue on build Version 18.06.1-ce-mac73 (26764)

[colbygk]

Can reproduce lack of inotify events using the above steps, though I also cannot get inotify events when performing the mount using -v either....

... ubuntu:latest ...

After the launch of the shell:

apt-get update && apt-get install -y inotify-tools

Mac OS X: High Sierra 10.13.6 Docker: Version 18.06.1-ce-mac73 (26764) stable

[mzaj]

I've started observing this issue after upgrade to OSX Mojave (10.14). I'm using Docker 18.06.1-ce-mac73 (26764) and everything worked fine before the upgrade.

[mzaj]

After switching to "2.0.0.0-beta1-mac75 (27117)" events seem to work again.

[davidolrik]

I'm seeing this problem again with Docker 2.0.1.0 (30090),

Touching a file on the host, does not propagate the events inside the container.

[mzaj]

I performed the update yesterday and so far everything works fine.

[govindrai]

@djs55 Issue seems to still be present in latest stable build. Facing this issue as of today: Docker version 18.09.2, build 6247962, Docker Desktop version 2.0.0.3. System: MacOS Mojave 10.14.3

Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        6247962
 Built:             Sun Feb 10 04:12:39 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 04:13:06 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[vaughnkoch]

I'm also seeing this issue with Docker for Mac 2.0.0.3, using the Node build of Alpine. I verified it with inotifywait -rm . inside the container, changing a file from a bash shell inside the container (triggers) and outside the container (does not trigger).

Container base: node:alpine

$ docker --version Docker version 18.09.2, build 6247962

$ docker info Containers: 8 Running: 7 Paused: 0 Stopped: 1 Images: 69 Server Version: 18.09.2 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 4.9.125-linuxkit Operating System: Docker for Mac OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 2.934GiB Name: linuxkit-025000000001 ID: UDQO:UPVW:W2XD:IDBJ:CH23:FTVR:64MU:FZVP:V5KV:YH4J:45NW:NKOK Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): true File Descriptors: 95 Goroutines: 133 System Time: 2019-05-04T16:06:42.4025166Z EventsListeners: 3 HTTP Proxy: gateway.docker.internal:3128 HTTPS Proxy: gateway.docker.internal:3129 Registry: https://index.docker.io/v1/ Labels: Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine

[KimiNewt]

I'm also seeing this issue, using -v on Docker version 18.09.2 on OSX 10.14.3, with a debian container. No events are propagated from the host.

[thaJeztah]

@KimiNewt could you try the steps I commented in https://github.com/docker/for-mac/issues/1802#issuecomment-483385012 to see if that works?

[dmitry-kostin]

It seems I have the same problem with 18.09.2 and 18.09.3 just checked this by installing back 18.09.1 and it works but doesn't on 18.09.2

I have a docker-composer file with following: volumes:

• ../../angular:/app
• /app/node_modules

don't sure for 100% if this is related to current issue but I have a feeling that yes

btw @thaJeztah, also I tried what you asked to try and i got the same output in all 3 versions

Only solution for me is keep using 18.09.1 ¯_(ツ)_/¯

[dmitry-kostin]

oh god, after hours of fighting i made it work, you may ask how? Well, hope this will help somebody, i got to install last stable version, remove it from docker ui via button, (I also pressed on all other buttons there) and installed again, I didn't even understand what was that, some nightmare..

ps none of my project or docker files were changed during this troubleshooting, so that was definitely some problem with docker for mac

[sanjams2]

GIven the comments above this seems to still be an issue? I'm at least not able to get file system events working with --mount or -v. I updated to the most recent macOS release as well as updated docker. I ran the suggested steps in #1802 and to demonstrate the issue

jsanders67 in /tmp/docker_shtuff
$ # setup
mkdir -p mount_event_demo
docker run -d --rm \
  --name filewatcher \
  -v $(pwd)/mount_event_demo:/container-path \
  alpine sh -c 'apk add --no-cache inotify-tools && inotifywait -m /container-path'
dc832f1911ad112cda8da8b018adcd7c677bf270bda4ef9a2f79cf99631e8e31

jsanders67 in /tmp/docker_shtuff
$ # do some file stuff
cd mount_event_demo
touch one two
echo I am one >> one
echo I am two >> two
mv one one-renamed
cp two two.bak
rm two.bak
mkdir -p ./some/sub/directory
touch ./some/sub/directory/hello-world
mv ./some/sub/directory/ ./foo
rm -r ./some

jsanders67 in /tmp/docker_shtuff/mount_event_demo
$ # inspect logs
docker logs filewatcher
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/1) Installing inotify-tools (3.20.1-r1)
Executing busybox-1.30.1-r2.trigger
OK: 6 MiB in 15 packages
Setting up watches.
Watches established.

jsanders67 in /tmp/docker_shtuff/mount_event_demo
$ # confirm inotify is working; do some file stuff in the container
docker exec -t filewatcher sh -c "echo some text > /container-path/in-container-test.txt"

jsanders67 in /tmp/docker_shtuff/mount_event_demo
$ # inspect logs again
docker logs filewatcher
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/1) Installing inotify-tools (3.20.1-r1)
Executing busybox-1.30.1-r2.trigger
OK: 6 MiB in 15 packages
Setting up watches.
Watches established.
/container-path/ CREATE in-container-test.txt
/container-path/ OPEN in-container-test.txt
/container-path/ MODIFY in-container-test.txt
/container-path/ CLOSE_WRITE,CLOSE in-container-test.txt

jsanders67 in /tmp/docker_shtuff/mount_event_demo
$ # sanity check files are actually in /container-path
docker exec -t filewatcher ls -la /container-path
total 16
drwxr-xr-x    6 root     root           192 Sep 10 17:58 .
drwxr-xr-x    1 root     root          4096 Sep 10 17:58 ..
drwxr-xr-x    3 root     root            96 Sep 10 17:58 foo
-rw-r--r--    1 root     root            10 Sep 10 17:58 in-container-test.txt
-rw-r--r--    1 root     root             9 Sep 10 17:58 one-renamed
-rw-r--r--    1 root     root             9 Sep 10 17:58 two

Docker info:

jsanders67 in /tmp/docker_shtuff/mount_event_demo
$ docker info # redacted for brevity
...
Server:
    ...
    Kernel Version: 4.9.184-linuxkit
    Operating System: Docker Desktop
    OSType: linux
    Architecture: x86_64
    CPUs: 2
    Total Memory: 1.952GiB
    Name: docker-desktop
    Runtimes: runc
    Default Runtime: runc
    Init Binary: docker-init
    containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
    runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
    Cgroup Driver: cgroupfs
    Server Version: 19.03.2
    Storage Driver: overlay2
     Backing Filesystem: extfs
     Supports d_type: true
     Native Overlay Diff: true

jsanders67 in /tmp/docker_shtuff/mount_event_demo
$ docker --version
Docker version 19.03.2, build 6a30dfc

macOS info: mac OS Mojave; Version 10.14.6 (18G95)

If people other than myself are also still having issues, then I think it's appropriate to reopen this issue.

UPDATE 1: as a work around for others, it does seem like writing to a mounted directory that is shared between containers does cause the file systems events to get triggered. So in the setup I describe above, you could run:

docker run --rm \
    -v $(pwd)/mount_event_demo:/tmp/outputdir \
    alpine \
    sh -c "echo Hello World > /tmp/outputdir/test.txt"

And you should see the events get picked up in the container that is watching the directory. Obviously the viability of this workaround depends on your specific use case

[johnhaley81]

I can confirm that I'm still having the issue.

[thaJeztah]

It looks like this may be because of the location used on the host.

Using the steps I wrote down in https://github.com/docker/for-mac/issues/1802#issuecomment-483385012, from within my home directory, I see the events showing up;

(1/1) Installing inotify-tools (3.20.1-r1)
Executing busybox-1.28.4-r1.trigger
OK: 5 MiB in 14 packages
Setting up watches.
Watches established.
/container-path/ CREATE two
/container-path/ MODIFY two
/container-path/ CREATE one
/container-path/ MODIFY one
/container-path/ DELETE one
/container-path/ MODIFY one-renamed
/container-path/ CREATE two.bak
/container-path/ ATTRIB two.bak
/container-path/ MODIFY two.bak
/container-path/ CREATE two.bak
/container-path/ ATTRIB two.bak
/container-path/ MODIFY two.bak
/container-path/ DELETE two.bak
/container-path/ CREATE,ISDIR some
/container-path/ CREATE,ISDIR foo
/container-path/ CREATE,ISDIR some
/container-path/ DELETE,ISDIR some

Doing the same, but from within /tmp, the events do not show;

(1/1) Installing inotify-tools (3.20.1-r1)
Executing busybox-1.28.4-r1.trigger
OK: 5 MiB in 14 packages
Setting up watches.
Watches established.

The /tmp directory should be shared with the Docker VM;

[igMoreira]

Apparently the issue is related to links in the host. I'm using High Sierra and /tmp is a link to /private/tmp. Just tried using the mount with /private/tmp/source-folder and the file events worked just fine in the container.

[thaJeztah]

good find; didn't think about that; yes, macOS creates those (wondering if Docker Desktop can work around that)

[rubensayshi]

Everything was working for a long time on macOS for me, up until recent, I think the latest macOS update has broken things.

If I mount ~/dummy then I get inotify events, but mounting /System/Volumes/Data/Users/ruben/dummy: I get no events, eventhough I do see the file changes...

[leetrout]

Confirmed not working in MacOS 10.15.2 with docker for mac 2.2.0.0 (42247).

[jemc]

I don't get events from mounting neither ~, ~/dummy, nor /tmp/dummy - I have tried all three - they work on Linux (without docker-machine), but in MacOS (with docker-machine) they don't.

Docker version 19.03.7, build 7141c199a2
docker-machine version 0.16.2, build bd45ab1

Note that I installed via brew (brew install docker docker-compose docker-machine)

[kinghuang]

@jemc That's not Docker for Mac. You can install Docker for Mac using the docker cask. I suggest removing the docker, docker-compose, and docker-machine brews first.

brew uninstall docker docker-compose docker-machine
brew cask install docker

[docker-robott]

Closed issues are locked after 30 days of inactivity. This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle locked