search

facultyai / faculty-sync

Synchronise a local directory with your Faculty platform workspace
https://pypi.org/project/faculty_sync/
Apache License 2.0
10 stars 1 forks source link

Test and document using sml-sync on Windows #21

Open pbugnion opened 6 years ago

pbugnion commented 6 years ago

We should at least test sml-sync on the Linux subsystem in Windows to see if we can get something working.

Whatever the result, there should be some section in the README on Windows support.

janfreyberg commented 6 years ago

Do we have a machine running windows 10?

pbugnion commented 6 years ago

Yes, in the engineering room. I believe the plan is also to get a Windows laptop.

We can now reproducibly install sml on Cygwin with OpenSSH and rsync. Syncing files work etc. Installing sml-sync is proving trickier because it relies on Paramiko, which relies on libsodium, which is quite hard to compile on Cygwin.

One possible option to make installing more reliable would be to remove paramiko and just rely on rsync and ssh. paramiko is notoriously quite hard to install. By not maintaining a persistent remote session, this would also improve reliability.

pbugnion commented 6 years ago

We currently use paramiko for:

  1. getting the last updated time for a file during the watch-sync process -- this happens every time a file needs to be synced (to avoid accidentally overwriting files the user has modified on the remote)
  2. checking whether the suggested remote directory exists during initial configuration
  3. listing all remote subdirectories of a given path (to populate the directory suggestion box during the initial configuation)
  4. making directories on the remote as part of the watch-sync process
  5. removing files and directories on the remote as part of the watch-sync process
  6. moving files on the remote as part of the watch-sync process

The only item here where latency is really important is 1. I think the right course of action is to try replacing 1. with shelling out to ssh and see if that materially impacts user experience. If it doesn't, we can press on and do all the others.

pbugnion commented 5 years ago

I just validated changing step 1 to use rsync rather than a permanent paramiko connection. This increases the time spent in the watch-sync loop from ~0.4 seconds for tiny files to about 1 second for tiny files. For larger files, the effect is negligible.

I've played around with this while doing actual gmaps development and the extra time doesn't seem to affect the experience, so I think we should remove paramiko everywhere. Besides providing windows support, it will also make sml-sync handle network disconnections much more gracefully, since we wouldn't maintain a persistent connection between the laptop and the SherlockML server.

janfreyberg commented 5 years ago

Sounds good to me; seems like a good decision and that seems like reasonable sync times.

Great idea!