Open lesteve opened 4 years ago
The command rsync -avz ~/folder-to-sync jean-zay:~/folder-to-sync --delete
may be slow when the folder you want to update contains many small files. This command is nice but does not scale well as rsync check every files in the destination folder before uploading any new change. A workaround in my case where I upload code is to add the option --cvs-exclude
to avoid copying .git
that contains many small files and clean up the folder once in a while to keep it at a reasonnable size (below 1-2 Go).
FYI #75 add automatic sync to the set of tips.
One usual way people work is to edit their file locally on their computer (they have all the tools they are used to, editor, etc ...) and synchronize on Jean Zay automatically (adding an explicit step like rsync or git push/pull is painful because you will forget it sometimes and when you do it will cost you a lot of time and frustration).
To synchronize a code folder (local -> Jean Zay)
inotify + rsync
The simplest and most generic thing we have found for automatic local -> Jean Zay sync is to use a script like this on your local computer (needs
inotify
andrsync
):If you are on a OSX you need to use
fswatch
rather thaninotifywait
, according to this the following command should work (not tested, comment if this issue if it does not work for you!).Editor extensions
Tip from @tomMoral if you are using Visual Studio Code (I also seem to remember there is an equivalent PyCharm functionality as well so this may be accessible in other editors as well):
You might consider using the vscode extension: https://marketplace.visualstudio.com/items?itemName=vscode-ext.sync-rsync
You can define a remote folder and sync your local project files to it. There is also a on save option that allows you to sync your project on save in vscode.
To view an arbitrary Jean Zay folder on your local computer
An alternative is
sshfs
with the right options to prevent being automatically disconnectedsshfs
can be useful for example to look at TensorBoard logs on Jean Zay with TensorBoard running on your local computer.For editing files, user feed-back seem to indicate that using
sshfs
to edit files is a bit laggy so not that useful in practice, if you have tweaks or suggestions to makesshfs
a more viable option for editing files, please comment in this issue!