Automatically keep DVCS repositories in sync whenever changes happen by automatically committing and pushing/pulling.
Set up desktop notifications (for these nice bubble-style popups when anything happens) and log into a Jabber/XMPP account specified in the config file.
Monitor a specific path (and its subdirectories) for changes with inotify. At the moment, only one path is supported and multiple script instances have to be run for multiple disjoint paths. This path is assumed to be (part of) a repository. Currently tested with git, but should support most DVCS (the config file allows to specify the DVCS commands called when interacting with it). Optionally, an [ignores] file is read with one exclusion pattern per line and files matching any of the patterns are ignored. This will typically be the .gitignore file already existing the git tree.
When changes are detected, check them into the repository that is being monitored (or delete, or move, etc.). It automatically ignores any patterns listed in .gitignore and the config file allows to exclude other directories (e.g. repositories within the main repository).
Wait for a configurable time. When nothing else changes in between, commit.
Wait a few seconds longer (again configurable) and, if nothing else is commited, initiate a push.
After the push has finished, send an XMPP message to self (that is, to all clients logged in with the same account) to notify other accounts of the push.
Furthermore:
Linux:
Mac OS X:
Recommended:
(on Mac OS X, see INSTALL_MAC file for detailled instructions)
[PREFERRED] PACKAGE INSTALLATION
MANUAL INSTALLATION
[on the server used to host the central git repository] $ git init --bare autosync.git
[on the first host using that repository]
$ cd ~ && git clone
[on each additional host]
$ git clone
Note that these are only examples. You can use arbitrary directories and repositories.
autosync.py [config file] # config defaults to ~/.autosyncfor Jabber login, there probably needs to be a
_xmpp-client._tcp.
when there are errors ERROR:pyinotify:add_watch: cannot watch ... on startup, it will either be an invalid file or directory name which can not be watched for changes, or the number of files a user may watch concurrently using the kernel inotify interface has reached the set limit. In the latter case, the limit can be changed by modifying the sysctl variable fs.inotify.max_user_watches and increasing it to a sufficient value (e.g. 500000).
Note that, when keeping changing media files (or other large binaries) in an automatically synchronized repository, it may grow quickly. In the current version, dvcs-autosync will never delete any history and keep all previous versions. This is intentional for documents and text files, but may be problematic for large binaries. I will try to address this problem in future versions, e.g. by integrating with git-annex (see TODO).
The XMPP push message already contains a parameter, namely the repository the push went to. Add another parameter to specify the repository in which the change happened so that others can try to pull directly from there, in case it is quicker. The main use case for this optimization is my standard one: the laptop sitting next to the desktop and both of them syncing each other's home directories. Going via the main, hosted server is quite a bit more inefficient than pulling via 1GB/s LAN....
Pulls and pushes can and should be optimized. At the moment, I take a conservative locking approach whenever a conflict may occur and performance is reasonable on my main work tree with ca. 16GB (cloned GIT repo), but not stellar. Specifically, actually implement the "optimized" pull lock strategy already described in the example config file.
Implement another option for synchronization besides XMPP (idea: a simple broadcast reflector on a single TCP port that could even run on e.g. OpenWRT, or re-use whatever the Sparkleshare server does).
Automatically adding some context to each commit message besides the automatic date/time would be useful for finding out why a change happened. Nepomuk anybody (just kidding, maybe, for now...)?
Allow to specify commit messages via popups. When ignored, use default commit message.
This is my first Python program that is longer than 100 lines. Please be easy on me with the patches, complaints and "what did you think, doing it this way?" messages. I have tried to comment wherever I found it necessary for my own understanding, but this is neither the best structured nor the most elegant program I ever wrote. Any hints for improving it are greatly welcome, and interoperability patches to work with Sparkleshare even more so. In the future, the two projects should definitely interoperate, which will come done to implementing each other's notification mechanism. My autosync Python script could then be used wherever headless operation might be required and/or Mono is not installed. I have tested it between three systems and, in this version, it works reasonably well. However, there does seem to be the occasional kink when editors go crazy on temporary file creation, renaming, deleting originals, etc. These might be races, but I don't know for certain yet. Additional test cases are more then welcome. This script should be fairly safe to try, considering that the worst it will do is add a few hundred commits to your DVCS repo and push them to the configured default remote. But, after all, what is the point in using a DVCS if you can't roll back any changes made by you or a buggy script (yes, I did have to do that a number of times while developing the manual inotify event coalescing to cooperate better with git add/remove/mv actions).
Rene Mayrhofer rene@mayrhofer.eu.org