slows down very much when ignored files are too many

[radsimu]

This happens in OS X. I have a build folder which is set to be ignored by unison. When that folder gets too big, unison syncing becomes extremely slow. Takes 3-5 minutes to finish one sync and the python unison-fsmonitor process consumes 50% of one cpu. So, as soon as a file is changed the unison-fsmonitor starts working for like 3-5 minutes and only after that, the syncing is performed. As soon as I clean up the build folder, syncing works fine. I have the latest unox version, just installed a few days ago. If you think this is a unison issue, let me know and I will file an issue with them instead

[EugenMayer]

Yes, that is one of the flaws of unox - you will not be able to fix that - its cause is due to the bad fs-event implementation of apple. macfsevents and watchdog are just working around this. Are you coming from docker-sync?

[radsimu]

nope. just trying to sync my workspace from a mac to a rhel vm that is hosted on an ubuntu.

[EugenMayer]

you probably have better results doing docker-for-mac + osxfs + then unison under linux ( inotify ). I do that with docker-sync "native_osx".

You basically mount your workspace into a docker-container and the use unison in there to sync it whereever you want. The interesting point is, that OSXFS is better in huge trees then unison and you can actually apply ignores under linux with unison inotify beside unison under linux does not need hardly any CPU

[EugenMayer]

That said, unox is limitted by what macfsevents / watchdog can get out of mac fs events, OSXFS events goes a level deeper. Especially with the new :cached / :delegated modes you will have a lot better results going over a docker-container then using native events under OSX

[hnsl]

Yeah, file watching can only get so good as the OS APIs allows it to be. Are you sure the build folder is really excluded? I remember that being tricky in unison.

[thule0]

+1 on slow updates detection. Why do excluded files even matter? If they are a problem, a watch can be set up to monitor whole directory, excluded or not, and when update will be triggered, it won't do anything.

Because of this issue, I'm running https://github.com/emcrisostomo/fswatch instead of unox to trigger unison (not integrated with the protocol, just fswatch -r -o $folder | unison $profile more or less), and it does not have the issue, it picks up updates almost instantly.

[BlinkyStitt]

I'm seeing 100% CPU usage with unox with syncing a very large directory with some large ignores on OSX. @roman-alexeev, can you share your profile that makes fswatch work? I'd like to see how that compares.

$ fswatch -r -o "$folder" | unison -auto -prefer newer "$local" "$remote"
Unison 2.51.2 (ocaml 4.07.0): Contacting server...
Connected [//tank//home/mgmt/mgmt -> //vacuum.local//Users/bwstitt/mgmt]
Looking for changes
  Waiting for changes from server                                       
Reconciling changes

local          tank               
         <---- changed    .git/index  
new file ---->            foo  

Proceed with propagating updates? [] y

Unrecognized command '3': try again  [type '?' for help]
Proceed with propagating updates? [] Unrecognized command '1': try again  [type '?' for help]
Proceed with propagating updates? [] ?

EDIT: nevermind. Looks like I also needed to add -batch to the unison command.

Doing it this way causes unison to restart each time a change is detected. I think that is still faster than unox. There is a major issue with using fswatch instead of unox though. Changing files on the remote system doesn't start a sync because unison isn't running there until a change is detected locally.

I noticed that the 100% CPU usage is happening while rls is running since rls is modifying a bunch of files that I have unison set to ignore. To work around this, I'm using https://github.com/rust-lang/rls/issues/622 to set the target dir to somewhere that unison isn't watching. This seems like a better fix than using fswatch over unox