codekitchen / fsevents_to_vm

forward OS X file system events to a VM, designed for use with Dinghy
MIT License
22 stars 10 forks source link

Fix nanosecond timestamps on high sierra #9

Closed ms-ati closed 6 years ago

ms-ati commented 6 years ago

On Mac OS High Sierra, the new file system (APFS) has changed file timestamps from second to nanosecond resolution. This has broken many dinghy-based development environments, please see https://github.com/codekitchen/dinghy/issues/269 for more details.

To become compatible with nanosecond timestamps, this patch does the following:

  1. Format times as ISO8601 with nanosecond resolution
  2. Via SSH invoke touch with '-d' option, which accepts ISO8601 format
  3. At startup install the GNU version of touch into Dinghy's VM

The last point is necessary because the default BusyBox touch is not itself compatible with nanosecond timestamps in the '-d' option. We copy the actual binary from the docker image for Ubuntu:16.04, which is verified compatible.

Please note that, completely separately, fixes were also needed in UNFS3.

ms-ati commented 6 years ago

When running with --debug for the first time:

$ bundle exec ruby exe/fsevents_to_vm start --debug --ssh-identity-file ~/.docker/machine/machines/dinghy/id_rsa --ssh-ip $(dinghy ip) ~
Checking if GNU touch already installed in Dinghy VM...
+ dinghy ssh gtouch --version
++ sh: gtouch: not found
++ exit status 127
Not installed.
Installing GNU touch from Ubuntu into Dinghy VM...
+ HOST_NFS_DIR=/Users/msiegel
++ mktemp -d /Users/msiegel/tmp/dinghy_tmp.XXXXXXXX
+ HOST_TMP_DIR=/Users/msiegel/tmp/dinghy_tmp.TAyMGFjH
+ mkdir -pv /Users/msiegel/tmp/dinghy_tmp.TAyMGFjH
+ docker run -v /Users/msiegel/tmp/dinghy_tmp.TAyMGFjH:/host_tmp_dir ubuntu:16.04 /bin/cp /usr/bin/touch /host_tmp_dir/
+ dinghy ssh sudo cp /Users/msiegel/tmp/dinghy_tmp.TAyMGFjH/touch /bin/gtouch
+ rm /Users/msiegel/tmp/dinghy_tmp.TAyMGFjH/touch
+ rmdir /Users/msiegel/tmp/dinghy_tmp.TAyMGFjH
+ dinghy ssh gtouch --version
+ grep 'GNU coreutils'
touch (GNU coreutils) 8.25
done.
Watching /Users/msiegel and forwarding events to Dinghy VM...
ms-ati commented 6 years ago

And then when running with --debug a second time:

$ bundle exec ruby exe/fsevents_to_vm start --debug --ssh-identity-file ~/.docker/machine/machines/dinghy/id_rsa --ssh-ip $(dinghy ip) ~
Checking if GNU touch already installed in Dinghy VM...
+ dinghy ssh gtouch --version
++ touch (GNU coreutils) 8.25
++ Copyright (C) 2016 Free Software Foundation, Inc.
++ License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
++ This is free software: you are free to change and redistribute it.
++ There is NO WARRANTY, to the extent permitted by law.
++
++ Written by Paul Rubin, Arnold Robbins, Jim Kingdon,
++ David MacKenzie, and Randy Smith.
Already installed.
Watching /Users/msiegel and forwarding events to Dinghy VM...
ms-ati commented 6 years ago

Hi @codekitchen, first of all, thank you for your input!

I am willing to commit to re-implementing this in terms of the Net::SSH. Totally happy to test out the idea that the /bin/touch from the dinghy-http-proxy might work, but since nanoseconds are not working inside that container, I had made the assumption (perhaps incorrectly) that the touch binary was compiled without nanoseconds support.

Totally agree that, if that touch binary is compatible with the host BusyBox and compiled with nanoseconds support, than it is definitely a better choice due to already being downloaded by dinghy.

codekitchen commented 6 years ago

Cool sounds good thanks. BTW I'm not sure what you mean by "since nanoseconds are not working inside that container", we don't ever call touch inside the dinghy-http-proxy container, just on the host VM (boot2docker/busybox).

ms-ati commented 6 years ago

BTW I'm not sure what you mean by "since nanoseconds are not working inside that container"

Thanks I may have skipped an import point there. I evaluated dinghy-http-proxy container as a target for executing the touch command there, since it will be running, rather than on the host VM, but couldn't get that to work.

codekitchen commented 6 years ago

Thanks I may have skipped an import point there. I evaluated dinghy-http-proxy container as a target for executing the touch command there, since it will be running, rather than on the host VM, but couldn't get that to work.

Ah, that makes a lot more sense then, OK.

codekitchen commented 6 years ago

Hm I don't think I was clear enough, sorry. What I mean is that if you SSH into the VM and run those sudo docker cp steps there, that'll copy the binary directly from the container into /bin/touch on the VM. So you can skip all the steps to create a temporary directory under the NFS mount and all of that.

ms-ati commented 6 years ago

Great idea! I'll try that.

ms-ati commented 6 years ago

@codekitchen Ok! This PR has been updated to remove any explicit dependency on Dinghy. It now works exactly as discussed above. Looking forward to your re-review :)

codekitchen commented 6 years ago

Sweet, looks good and everything seems to be working great when I pull it down locally. Thanks for doing all that. I'll cut a new release of Dinghy today or tomorrow.

ms-ati commented 6 years ago

Nice! @codekitchen could I impose on you to do the following additional steps:

  1. Release new Dinghy that depends on this fsevents_to_vm
  2. Update homebrew-dinghy to require the new Dinghy
  3. Add a highly visible note to the README to inform people that it now supports nanosecond timestamps on High Sierra, and please run brew update; brew upgrade dinghy to fix it
  4. Close https://github.com/codekitchen/dinghy/issues/269 with an informative message outlining the final state (for future folks reading it)?
codekitchen commented 6 years ago

Yes, updating homebrew-dinghy is part of releasing a new Dinghy version. Thanks. It'll be in the CHANGELOG.