davidbannon commented 7 years ago

I am trying to understand how the Sync system works and think it makes sense, if I discover enough to be useful, for it to end up on the Wiki. Even if I abandon my project, someone may need to understand how it works. Below is my first draft but i have to point out, at this stage, I have serious question marks against two issues.

Tomboy's Sync Policies

We like to think that the Ancients had a secret knowledge that we don't have anymore. That does seem to be the case with Tomboy design. The Masters Of Old Tomboy, MOOT, worked very hard on a number of points that have made Tomboy as robust and reliable as it is. Sadly, no one thought to summarize those MOOT Points, we are left to read back over old discussions or reverse engineer the code. This page is about just that.

Sync Model.

Broadly, with sync, we have a server and a client. The client is the bit of Tomboy the user sees on his or her desktop. The server may be a Tomboy-web-sync system such as Rainy or Grauphel or may just be a file repository that is somehow shared (Google Drive, Dropbox etc) between systems we sync with. Much of the policy does not care. Here I talk about File Based Sync, tomboy-web-sync is probably similar with a network layer between. And obviously, it has server code to implement this policy.

Every note has a (hopefully) unique ID that identifies it and is also used as its filename. Here we will just call that an ID. A Server may be connected to from multiple clients, any of whom may add, remove or edit a note. It seems to be assumed a client can only ever connect to one server without reconfiguration. The Sync Repository has a file, manifest.xml that lists all IDs it believes, at the time, should be in a client. It lists a revision number [0..x] for the ID that refers to a directory that contains the most current revision of the note. All revision directories are under a directory called '0', I don't know why nor do I know if that zero can ever change. Thats an outstanding issue. The revision directories also contain a backup manifest file that could potentially be used to recover from an error.

At the client end, there is also a manifest.xml file (in eg $USER/.config/tomboy/.). It contains a last sync date and a server ID. It also lists IDs and revision numbers of notes it has sent to or received from the Server, updated at sync time. Between Syncs, the ID of any locally deleted note is added. Thats how the sync process can tell the difference between a new note introduced to the server from another client and a note that was deleted from this client after the previous sync. I think it makes sense to only list deleted notes that have previously been synced, unnecessary to list ones that were created and then deleted between sync events ? Unnecessary but does no harm ?

There are a number of situations that can happen, in no particular order (remember, its intended to be a REST system) -

CaseA - A client makes a new note then syncs. We check that it realy is a new not, ie not present in local manifest. The new note is added to Sync Repo. It's ID now appears in Sync Repo /manifest and a copy of the note is in /0/$RevNo/.
CaseB - A client deletes a note (previously synced or not), then syncs. The client moves the note to ~/BackUp/. When syncing, it may observe that the server knows about a note it does not have, so, check local manifest, ah, its been deleted. Tell Server to delete it from whatever previous revision directory it was in. Remove reference to it from local manifest.
CaseC - A client, when syncing, gets told about a note it does not have. A bit similar to above but this time the note is not in the deleted list (local manifest) so pulls it down and adopts it locally.
CaseD - A client, when syncing, has a note that has been deleted by another client. This is initially similar to case A. We'll assume CaseB ran to completion. The server does not have the note, so before the client kindly offers to upload it, the client checks to see if it has been uploaded or downloaded before by checking the local manifest. If its there, it has been deleted by another client, move to Backup, effectively, delete it locally
Pretty obvious how we deal with updates to existing notes, as long as one or the other dates back to the client's last sync. But if both copies have been updated since last client sync date, thats a clash.

Ref - https://bugzilla.gnome.org/show_bug.cgi?id=321037 https://wiki.gnome.org/Apps/Tomboy/Synchronization/IRCLog18May2007 https://wiki.gnome.org/Apps/Tomboy/Synchronization/Brainstorming "Many weird situations can be avoided by performing an update in three distinct steps: deletes, then updates, then adds." repeated again and again. But not what happens in the code.

joshp23 commented 7 years ago

Thank you for writing this

christophra commented 5 years ago

Thank you for this explanation! I had a sync go awry (through no fault of Tomboy, ownCloud had to help) which ended with a bunch of notes "deleted" from the manifest. Do you know a way to simply roll that back to a previous revision? Edit: After a bit of poking around I think I can answer my own question. It was just a matter of quitting Tomboy, copying the revision's manifest to the root of the sync directory, making sure that the notes were still available in the directories corresponding to the revisions listed in that manifest (in my case that was simple since the sync error hadn't touched them), then starting Tomboy again and synchronizing.

davidbannon commented 5 years ago

Chrisophra, thats a great outcome. Well done.

I could not have helped you anyway. As part of the tomboy-ng development, I made a rough and ready tool that can repair a file based sync repo - but it does not know how how to use eg OwnCloud. But it sounds like your repo was intact - great !

Davo

christophra commented 5 years ago

Actually I think that tool would indeed have helped here! It's still a file-based sync, only that OwnCloud stopped and didn't update the files before Tomboy synced. So all that happened was that Tombboy saw an old root manifest.xml and then created a new revision based on it. But as far as I understand, the .note files are never deleted or moved, for a new revision you just create a new copy, right? That must be why there were no changes to the data.

On Tue, Feb 19, 2019 at 11:31 PM davidbannon notifications@github.com wrote:

Chrisophra, thats a great outcome. Well done.

I could not have helped you anyway. As part of the tomboy-ng development, I made a rough and ready tool that can repair a file based sync repo - but it does not know how how to use eg OwnCloud. But it sounds like your repo was intact - great !

Davo

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tomboy-notes/tomboy/issues/75#issuecomment-465337991, or mute the thread https://github.com/notifications/unsubscribe-auth/APiEXdufecqCgOnvjtzhupY5RGYguVmXks5vPHsygaJpZM4P5ojS .

davidbannon commented 5 years ago

Yes, the tool I have would have to be run on the Server - that way it could see the files OwnCloud put in place. But I don't have any experience in poking around the files at that end, its likely they are stored there in the format but not guaranteed. Maybe I need to polish that tool up and publish. It was made for one particular user with a particular problem, might be usefully expanded to a generic tool.

davidbannon commented 5 years ago

Oh, one more thing, both Tomboy and tomboy-ng make copies of notes before deleting or overwriting (during sync). tomboy-ng has some tools to bring those files back if needed, with Tomboy you need to do it manually.

tomboy-notes / tomboy

Documenting The Sync Policies #75

Tomboy's Sync Policies

Sync Model.