Switch from fsevents to watchdog

[botimer]

This change swaps out the base fsevents API for watchdog. It's not all that well tested, but the basics are working.

In my testing on 10.11, fsevents was causing indeterminate delays of up to ~10 seconds for events to be triggered. Using watchdog results in the expected "immediate" behavior.

[hnsl]

Sorry, for being slow. I will merge if you fix the conflict. If you say it works it works. The code looks good. :)

[botimer]

No problem. I haven't tested this on Sierra yet, so that's probably worth a quick check. I don't expect any issues, but better safe than sorry.

[hnsl]

You still have a conflict. :)

[botimer]

Thanks for checking back. I have resolved the conflict, though testing on Sierra is still outstanding.

[EugenMayer]

@botimer how are your results with watchdog and what are the reasons to switch?

As far as i can measure it, watchdog needs more CPU and has a much higher latency to detect changes - sounds like a loose-loose? I am right now thinking about reverting back, see #17 - could you join the discussion and share your thoughts?

[botimer]

@EugenMayer I was seeing multiple second delays and needing to double-save for detection before using watchdog. Results could definitely vary. I haven't been actively using this in a while, so I don't have current data (10.12 on SSD).

If you are seeing better performance with fsevents, maybe it's either better or I was seeing something anomalous. It seems like you have taken measurements to support your conclusion. I'd say go with what works for you.

[EugenMayer]

How large is the codebase you are watching? 10,30,50k files?

Would be nice to know under which circumstances things can vary

[botimer]

At a glance, including the .git directory, one of these is ~4k files, ~1200 directories.

On Fri, Apr 28, 2017 at 11:22 AM, Eugen Mayer notifications@github.com wrote:

How large is the codebase you are watching? 10,30,50k files?

Would be nice to know under which circumstances things can vary

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hnsl/unox/pull/8#issuecomment-298027787, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJtK3VkAOEan8oC9DWoXIkLFkdx5czLks5r0gQ9gaJpZM4H48Ia .

[EugenMayer]

Ok so that's one of the huge differences, we are working with 20k+ .. lets see if I can benchmark it properly, thank you

[donnykurnia]

@EugenMayer one thing that I noticed from the new unox with watchdog is that the sync daemon continued to work after my Macbook wake up from sleep. The old unox with fsevents will always crash, so I need to stop daemon and start it again. For the past few days, I haven't restarted the docker-sync-daemon even after a lot of sleep by idle and closing MacBook lid then wake-up and resume my work.

Unfortunately, right now I don't have spare time to test this effect further.

My project is a Rails 4.2 application with around 88.000+ files, most of these are gems files inside vendor/bundle folder. Application files only around 16.000+ files. I'm using docker-machine with virtualbox in my MacBook 13" 2012 using El Capitan OSX.

Additional note: I don't see slowdown or any CPU spike when using new unox + watchdog. The sync work just fine, even better than the old unox + fsevents. Part of the cause might be right now I have more free space after deleting unused files. I only use 73% capacity of my Macbook main harddisk.

[EugenMayer]

Thank you for sharing this @donnykurnia - did you had any performance decrease with watchdog? It seems the amount of files do slow it down enourmously - does it do the same for you?

Do you have any excludes in your docker-sync.yml which does remove some of those? For unox though, that does not matter, excludes are for unison only, unox watchs the whole FS anyway

[donnykurnia]

@EugenMayer I don't see any performance decrease. Here is my docker-sync.yml file:

version: "2"

options:
  max_attempt: 30
  # verbose: true
syncs:
  bundle-bundle-sync:
    src: './vendor/bundle'
    dest: '/app/heroku/ruby/bundle'
    sync_excludes: ['.DS_Store', '._*']
    sync_host_ip: '192.168.101.100'
  bundle-app-sync:
    src: '.'
    dest: '/app/user'
    sync_excludes: ['vendor/bundle', 'vendor/cache', 'tmp/cache', 'public/assets', '.git*', '.DS_Store', '._*', '.docker*', 'docker*yml', 'Dockerfile*', 'notes', 'data']
    sync_host_ip: '192.168.101.100'

The folder that changes much is .git folder and even when I do many git related tasks, I don't see any performance hit with the new unox.