File system change events broke after upgrade

villasenor commented 7 years ago

Expected behavior

inotify/fsevents provides files system change events from mounted directories inside the docker container

Actual behavior

file system change events are not propagated to the container

Information

File system change events have randomly stopped working. It may be related to the most recent Docker for Mac stable update. Note that uninstalling and reverting to an old version does not fix this. Also, recreating containers on the new version does not fix this.

Installs of the latest Docker for Mac stable version on a brand new Mac do not exhibit this behavior. This behavior only occurs on Macs that have previously had an older version of Docker for Mac, but I’m not exactly sure.

macOS: version 10.11.6 (build: 15G1510)
logs: /tmp/BCD22BE2-D5C8-43A6-A3EF-B1CE3BE78EA5/20170630-113038.tar.gz
[OK]     db.git
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

Diagnostic ID: BCD22BE2-D5C8-43A6-A3EF-B1CE3BE78EA5

Steps to reproduce the behavior

docker run -v /Users/myuser/dockertest:/opt/dockertest -it alpine:3.5 sh
apk update
apk add entr
ls *.txt | entr echo "changed!"
Back on the host mac, add a txt file or modify an existing txt file in /Users/myuser/dockertest
Notice that entr does not add another "changed" line, as it should. Indicating that file system change events are not propagating. This used to work.

villasenor commented 7 years ago

After reading through a bunch of these issues, it seems that one of you might be able to help with this?This problem is slowly creeping through our organization and breaking build processes. I can't narrow down exactly what is causing it yet. I'm sure you all have different specialties so just chime in if you have anything you can offer. I would really appreciate it! @dsheets @djs55 @yallop @thaJeztah @dnephin @tonistiigi @nishanttotla

villasenor commented 7 years ago

@jasonbivins any update on this?

thaJeztah commented 7 years ago

I can reproduce this (but never used the entr tool, so don't know if it worked before on my machine :smile:; I had to slightly alter the reproduction steps;

mkdir -p repro-1802 && cd repro-1802

# had to have a file, otherwise `entr` exited when starting
echo "hello" > foo.txt

docker run -v /Users/sebastiaan/projects/test/repro-1802:/opt/dockertest -it alpine:3.5 sh
apk add --no-cache entr
cd /opt/dockertest
ls *.txt | entr -p echo "changed!"

Changing the foo.txt file on the host did not trigger changed!, but updating the file from within the container did.

Note that uninstalling and reverting to an old version does not fix this.

This behavior only occurs on Macs that have previously had an older version of Docker for Mac

Have you also tried

doing a factory reset (this will errase the docker for mac storage, so you'll loose containers, and volumes)
are both machines that do and do not have the issue the same storage driver ("overlay2")? Older versions of docker for mac used "aufs"
are OS X versions exactly the same on machines that work and don't work?
are machines that work and don't work using the exact same version of the image? (i.e., if you docker inspect -f '{{.RepoDigests}}' alpine:3.5, do they show the same sha256?)

villasenor commented 7 years ago

Thanks for taking a look at this @thaJeztah! I'll go do some digging on the storage driver, macOS versions, and sha256 hash of the images. Tried a factory reset though and still having the problem.

thaJeztah commented 7 years ago

I think @dsheets is looking into this as well

fertapric commented 7 years ago

I had experienced the same issues. However, in my case, a factory reset fixed them, but they appeared again eventually.

My Docker version is 17.06.0-ce-mac19 (18663) (stable channel)

dsheets commented 7 years ago

@delmartechdude entr seems to listen for IN_MODIFY events but does not translate them to its BSD-like internal NOTE_WRITE events. See https://bitbucket.org/eradman/entr/src/ed4136a57481e7a7469cda340780066bf4e48a42/missing/kqueue_inotify.c?at=default&fileviewer=file-view-default#kqueue_inotify.c-142. I'm not sure how entr has ever worked for you with Docker for Mac shared file systems. I may be misunderstanding how the tool works but it also seems to exhibit bizarre behavior when files it is monitoring are moved or deleted regardless of the file system:

# ls | entr -p echo "changed" &
# rm bar
# entr: cannot open 'bar': No such file or directory

[1]+  Done(1)                    ls | entr -p echo "changed"

Fixing #896 may help as IN_CLOSE_WRITE would be propagated in addition to IN_MODIFY. Perhaps a tool like inotifywait would serve you better until entr is improved? Or perhaps a friendly bug report/PR to entr would help?

I've tried on Docker for Mac 17.03.1-ce-mac12 (17661) and I experience the same behavior from entr. If you believe this is a regression, could you please provide a successful use of entr on osxfs?

villasenor commented 7 years ago

Thanks for looking into this @dsheets! Working on gathering more info for you as requested. Stand by!

docker-robott commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

villasenor commented 6 years ago

@dsheets we ended up abandoning Docker for Mac and doing our own Vagrant + Linux + Docker Daemon with Unison sync. But I would like to see this fixed eventually. Can we reopen this? I don't have time to gather more info but this still warrants additional investigation. Also, I used entr as an example, but lysncd was actually being used originally which uses inotifywait.

villasenor commented 6 years ago

/remove-lifecycle stale

thaJeztah commented 6 years ago

ping @djs55

docker-robott commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

villasenor commented 6 years ago

/remove-lifecycle stale /lifecycle frozen

mr-nano commented 5 years ago

No updates on this? Isn't this like a show-stopper for development use case?

govindrai commented 5 years ago

Completely agree. I can't develop on my mac if I have to keep having to build images after every change. Requesting updates.

thaJeztah commented 5 years ago

Wonder if anyone reported the issue with the entr maintainers.

FWIW, I tried @dsheets suggestion to use inotifywait;

Create a directory for testing;

mkdir testing
cd testing

Run an alpine container that bind-mounts the current directory to /container-path inside the container; The container starts, installs inotify-tools, starts inotifywait and prints all events it received in a loop.

The container is running in the background, but we can view its output through `docker logs

docker run -d --rm \
  --name filewatcher \
  -v $(pwd):/container-path \
  alpine sh -c 'apk add --no-cache inotify-tools && inotifywait -m /container-path; while read path action file; do echo $file; done'

On the host (Mac), create, edit, rename, and remove some files;

touch one two
echo I am one >> one
echo I am two >> two
mv one one-renamed
cp two two.bak
rm two.bak
mkdir -p ./some/sub/directory
touch ./some/sub/directory/hello-world
mv ./some/sub/directory/ ./foo
rm -r ./some

Check the logs of the container;

docker logs filewatcher

fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/1) Installing inotify-tools (3.20.1-r1)
Executing busybox-1.29.3-r10.trigger
OK: 6 MiB in 15 packages
Setting up watches.
Watches established.
/container-path/ CREATE two
/container-path/ MODIFY two
/container-path/ CREATE one
/container-path/ MODIFY one
/container-path/ DELETE one
/container-path/ MODIFY one-renamed
/container-path/ CREATE two.bak
/container-path/ ATTRIB two.bak
/container-path/ MODIFY two.bak
/container-path/ DELETE two.bak
/container-path/ CREATE,ISDIR some
/container-path/ CREATE,ISDIR foo
/container-path/ DELETE,ISDIR some

akauppi commented 3 years ago

~~Factory reset worked for me (Docker Desktop 4.0.0). Thanks, @thaJeztah.~~

Unfortunately that was pre-mature. It worked for a while, but in the same evening, no more.

I am going to build a hack-fix as described here.

akauppi commented 3 years ago

Turns out, one should check whether the tool not hearing file system events is using Chokidar.

And if so, read this:

It is typically necessary to set this (polling) to true to successfully watch files over a network, and it may be necessary to successfully watch files in other non-standard situations

      - CHOKIDAR_USEPOLLING=true

docker / for-mac