tomboy-notes / tomboy-ng

Next generation of Tomboy
MIT License
389 stars 38 forks source link

Synchronization support tomboy-ng <-> Nextcloud Notes #199

Closed monperrus closed 5 months ago

monperrus commented 4 years ago

Problem to solve: having a fully working setup with Linux desktop program / server side code / Android app for taking and synchronizing notes.

Tomdroid is dead, and one of the best Android app for self-hosted note taking today is Nextcloud + Nextcloud notes. So a potential solution is :

Any hint on how to proceed would be appreciated.

davidbannon commented 4 years ago

Sigh, it has to be JSON doesn't it ? Hope you are better at JSON than me.

I'll have a read of that API but it looks like it might be a lot better documented than the Tomboy model was. Could be good ...

Davo

monperrus commented 4 years ago

Yes, this is JSON. I agree that the documentation of https://github.com/nextcloud/notes/blob/master/docs/api/v1.md is nice!

Looking forward to your thoughts.

davidbannon commented 4 years ago

OK, had a quick and very superficial read.

The nextcloud note attributes seem to map fairly well to tomboy, only exception is favourite, maybe thats "open on start" but I don't think so. So, we'd need to have some lookup table or modify the tomboy note format, both ugly. The note ID is not going to be a 1:1 transfer, tomboy uses GUID and nextcloud has just a int. So, again, some mapping. Maybe thats version 2.0 ? The Title and Timestamp looks easy.

The API looks like it it covers what we need, simple Gets and Puts.

The tomboy model assigns a Sync ID to make sure it is talking to the right repository, maybe we can overlay the username for that. You would assume a nextcloud would have only one note repository ?

I think I would take the same approach here that I took with Tomdroid, I would not consider Nextcloud as just another note in the sync model. Nextcloud does not, and cannot easily, keep the necessary metadata. So we need to keep it locally, per connection. We end up with one tomboy-ng node that syncs to Nextcloud in a one to one sync. That tomboy-ng node can of course also sync to other tomboy nodes using its normal sync model.

Just to be clear what I mean, so you can tell me if I am being silly -

Suppose I have a Linux laptop, a Linux desktop, occasionally reboot my laptop into Windows. And an old Mac I use to run some tests. All of these systems are part of the 'normal' tomboy file sync.

But on my Laptop I can also do a 1:1 sync to Nextcloud so that my Phone and Tablet can also see my notes.

I cannot do that 1:1 sync from any other system than the Laptop but because the Laptop auto syncs to my file repo and so do the other systems, its all in sync.

We would translate the Tomboy markup to md at sync, again, we'd loose some data but I think it would be acceptable.

What do you think of that model ?

I believe its pretty easy to implement.

Another approach might be to make a Nextcloud note that holds the meta data. As long as user did not mess with it, it would work, I think. Risky IMHO.

Davo

davidbannon commented 4 years ago

monperrus, have a had a better read and a think about the issues. A bit rambling but enough for you to see that there is issues we have to solve. One of the biggest problems with syncing is knowing which files have changed and which have not. Thats certainly the issue here.

If we download a list of notes on nextcloud, we get a noteID and a timestamp. Neither are particularly tomboy compatible, the noteID is an integer, the time stamp is a Unix one, so only one second resolution, Tomboy uses seven decimal places of a second. That's actually important, the chances of a note being edited at two places in exactly the same second is small, in the same 100 nS is negligible. But we will have to work to the second !

The Nextcloud noteID is an integer, presumably starting at 1 (or 0). So, if you have to merge two sets of notes, or move from one NextCloud to another, we cannot guarantee that the noteID is unique. Tomboy uses a GUID, unique (enough). Once a note has its GUID, Tomboy does not care about the actual format so we could, perhaps clear the last six digits and put the Nextcloud number in there, with some marker to show what we have done.

Usual Tomboy file name -

e8cb904d-2c4b-4ab7-8d93-3d353fcbdda0.note

A note derived from NextCloud, say, 177 -

e8cb904d-2c4b-4ab7-8d93-3d353-000177.note

That now allows us to recognise this note as one we should compare to the Nextcloud note number 177.

However, when I send a Tomboy created note up, say -

627e0bde-99ca-4377-9c73-92dcca942680.note

Next cloud will change its ID to, say, 178 and at the next sync, we have no way to tell we need to compare it to the local 627e0.... If the POST/Note API returned the newly assigned Nextcloud ID then we could maintain a map of the IDs used at either end. But we'd need a process to remove entries relating to deleted notes etc. Do-able but messy.

The Tomboy sync model stamps a SyncRevision number on every transaction and a list of all the current notes and their latest revision. That works if the "remote" is happy to maintain that data for us. I see no option to do that on NextCloud Notes.

In my Tomdroid sync, I rely a lot more on the LastChangeDate, I read the last change dates of all notes in Tomdroid and bring it back to tomboy-ng to compare. We can do that but only to one second resolution....

Deletions on NextCloud (assuming we solve the above) If we receive a list of notes from the Nextcloud server and we notice we have one locally that is no longer there on Nextcloud, we assume its been deleted up there and should be deleted here. Thats OK.

Deletions on Tomboy Notes deleted locally that have been previously synced have their IDs recorded until the next sync, then the NextCloud is told to delete them. There is an API for that purpose.

Need for ServerID The deletion models require a guarantee that we are talking to a system we have synced to before. Else we get a real mess on our hands. So, both the Tomboy and Tomdroid sync models stamp an ID when a new sync repo is established, it asks about that ID when making any subsequent connections, if they don't match, do not proceed ! That ID, or some sort of token, really needs to be stored at both ends !

Note Title is not useful in identifying notes, user's change the title all the time.

A lot of this could be solved if we could find a way to store some metadata on Nextcloud. Could we, for example, just have another document there, created by the sync process ? ( Or, better still, is there any possibility we can update that document when a note is changed/deleted on Nextcloud ? I understand, a lot harder.) But just a nextcloud doc we can upload and download outside of the note system is, perhaps, what Nextcloud is all about ?

Sorry to be so picking, I am concentrating on the problems so, once we have solved them, its full steam straight ahead !

Note: if I was to redesign a new Tomboy schema, I would include a new GUID or maybe a revision number at every edit, that way I'd always be able to tell if the remote one was the same as the local.

monperrus commented 4 years ago

Thanks a lot for the comprehensive analysis.

The metadata problem is related to the file format.

If we store the note in Markdown on Nextcloud, we can use a YAML frontpage to store the metadata.

davidbannon commented 4 years ago

OK monperrus, that sounds hopeful !

Are you suggesting you could return a reproducable UUID (that you have allocated in Nextcloud) when the GET /NOTES call is made ? If so, it would all become quite easy !

I am currently concentrating on establishing an Ubuntu PPA and possibly getting tomboy-ng into debian but when thats sorted, I will build a nextcloud set up and have a play ....

Davo

monperrus commented 4 years ago

I am currently concentrating on establishing an Ubuntu PPA and possibly getting tomboy-ng into debian

Super nice!

but when thats sorted, I will build a nextcloud set up and have a play ....

Excellent, looking forward to it.

monperrus commented 4 years ago

Are you suggesting you could return a reproducable UUID (that you have allocated in Nextcloud) when the GET /NOTES call is made ?

There is already one, the id field:

[
    {
        "id": 76,
        "modified": 1376753464,
        "title": "New note",
        "category": "sub-directory",
        "content": "New note\n and something more",
        "favorite": false
    }, // etc
]

Am I missing something?

davidbannon commented 4 years ago

"id": 76,

The problem is that the ID is not unique. If someone syncs to their Nextcloud and later re-installs nextcloud or syncs to another install for some reason (and people do do that all the time !) they come across that same number again, because its got the same ID, its assumed its the same note. So, it overwrites it. Every Nextcloud Notes install will have a number 1, a number 2 and so on.

Truth is, Nextcloud Notes needs to assign a unique ID to every note even if it does not sync with Tomboy !

Davo

monperrus commented 4 years ago

Do you mean not unique per user? per server?

davidbannon commented 4 years ago

I mean it needs to be unique. Unique per server is not enough, suppose someone moves to another server that already has, perhaps one note. Unique per user is not enough, what happens when two users swap notes? Users send notes between them, there are, perhaps ten Tomboy format notes in the tomboy-ng install kit.

If all you want to do is 'import' or 'export' from one platform to another, then, its easy, no need to worry about unique, but if we are going to track them over some sync, edit, sync, delete cycles, we have to be sure we are talking about the same note.

Davo

monperrus commented 4 years ago

I understand. What about putting the unique id as a metadata?

davidbannon commented 4 years ago

Nine days ago I said

"A lot of this could be solved if we could find a way to store some metadata on Nextcloud. Could we, for example, just have another document there, created by the sync process ? ( Or, better still, is there any possibility we can update that document when a note is changed/deleted on Nextcloud ? I understand, a lot harder.) But just a nextcloud doc we can upload and download outside of the note system is, perhaps, what Nextcloud is all about ?"

Davo

monperrus commented 4 years ago

Hi Davo,

The solution I proposed is to store the note in Markdown on Nextcloud, and to use a YAML front matter to store the metadata.

davidbannon commented 4 years ago

You did indeed. And I thought that was a great idea. But you followed it up by saying -

There is already one, the id field: ... "id": 76,

Now, the way I understand it, everyone with more than 76 notes in their repository will have a note with ID = 76. So, IMHO, thats not unique. If you can replace the ID field with a UUID/GUID then it would be unique, that would be cool, we would be underway. But if the ID is allocated by ++(previous_ID) then we do not have a unique ID.

Just about every programing language these days has a library function to generate a UUID, its a reasonably clearly defined way to produce a (almost) guaranteed unique string.

I hope this is clear, we seem to be going around in circles here I am afraid.

Davo

monperrus commented 4 years ago

AFAIU, we can have two separate identifiers:

Makes more sense?

davidbannon commented 4 years ago

That sounds great ! I assume you can include the "logical id" in the initial REST response, along with the physical ID ? That would work. I don't necessarily need to use the logical ID when manipulating notes, as long as I have seen a confirmation that the mapping between them has not changed.

I'll need to do some thinking about how we handle deleted notes. When I see a LID (Logical ID in your terms) in Nextcloud, that I don't have locally, its either a new one that I have not seen before (action=download) or one I have seen before but decided to delete locally (action=remote delete). With my Tomdroid sync hack, I keep a one to one sync (and a list of deleted notes) from one instance of tomboy-ng to one Tomdroid phone. I think that may be better than making the Nextcloud Notes one participant in a Tomboy file sync but it will need some work to be sure.

Anyway, if we have a transport model worked out, a sync policy can be worked out too !

Davo

monperrus commented 4 years ago

I assume you can include the "logical id" in the initial REST response, along with the physical ID ?

I'm not a Nextcloud developer, I don't know whether we would be successful in pushing this change.

Is that required for our use case?

davidbannon commented 4 years ago

Yes, I think its necessary. Maybe we might get away with a having a (nextcloud?) document that maps UUID to NCNoteID but it would be fragile. Lots of things that could break it that we may not think of.

For example, and a test you can do perhaps ? Make a new note, its gets assigned, say, number 44, can you see its number ? Delete it immediately and create a another one. Does it get ID 44 or ID 45 ? If it gets 44, no, we cannot work like that. If it gets 45, maybe. I'll have to do some work to map out the risks.....

Davo

monperrus commented 4 years ago

Does it get ID 44 or ID 45 ?

I don't know.

If it gets 44, no, we cannot work like that

I am not sure, thanks to uuid metadata

davidbannon commented 4 years ago

Another long answer I am afraid. I don't think its hopeless but there are some serious downsides.

I don't think you can associate a note with a particular UUID if we cannot add it to the note's own meta data. Maybe I am wrong ? You understand what happens inside Nextcloud far better than I do so, perhaps you have a plan that works ? Can you explain it please ?

In my mind, if the UUID is not part of a note's metadata then we have to have some sort of lookup table. - ... ID=44 UUID=xyz ...

However, if the act of deleting that note and making a new one that uses the same ID does not automatically update the table, then we cannot tell it has happened, it looks like an edit. So, the process is perhaps -

Maybe thats an acceptable departure from the Tomboy model. It means for example I cannot backup the old, deleted content as per Tomboy practice. If there is a sync clash, in tomboy-ng I show a diff of the two notes, thats going to be a messy diff to read where everything has changed. If someone does a restore from snapshot, again unexpected results because we are reusing the original UUID, 'xzy'. Note creation date will be the time when the very first ID=44 was seen by tomboy-ng, that ID may well be reused many, many times with new notes but each one will still show the quite incorrect note creation date of the very first note to be assigned that ID.

I suppose we could take the view that a sync with NCNotes is not a normal Tomboy sync, as long as the end user understands there are times when unexpected results will occur.

I understand you don't think there is any prospect of getting the NCNotes people to add a UUID and/or a creation date ?

Davo

monperrus commented 4 years ago

Hi Davo,

Here is my understanding of the scenario (in bold my explanations)

--Martin

davidbannon commented 4 years ago

Right, I think we are focusing in here.

Next, the Nextcloud user deletes the note and create a new one, also assigned ID=44

If the user creates the note on Nextcloud directly, there is no UUID. Because there is no UUID, tomboy-ng understands that 1) a new UUID must created 2) it must update its internal lookup table for ID=44

No, there is still an entry in the table for ID=44 : xyz, no one has deleted it. tomboy will not not be able to detect that the note has been deleted and reissued. So, will use the old UUID. (different result if user sync after every delete).

If the user creates the new note on tomboy-ng, tomboy-ng knows that how to update its internal lookup table for ID=44.

Yes, in that case, tomboy-ng will know to remove to old entry for 44 from the table. It will request a new note on NextCloud (hmm, does NextCloud tell us the ID ? Thats probably another problem, if it does not get told the ID of the new note, it cannot add an entry to the table.).

I am starting to think an export / import filter is practical but not real syncing. No prospect of interesting the NextCloud developers in adding a UUID to a note's metadata ? It would be useful for a lot more applications than tomboy-ng.

Sorry I am being so negative !

Davo

davidbannon commented 4 years ago

OK monperrus, I have built a nextcloud install (wow, finally found a use for Snaps!) and have some more data. Interesting.

Note IDs are not allocated sequentially, indeed they jump a lot more than one per note, I have run up some six notes and each one was (substantially) greater than the previous one. The sequence was -

136 155 (deleted) 193 212 (shutdown and restart) 339 Maybe its elapsed time based ?

So, if they never go back, that does make them kind of unique, doesn't it ?

Not unique across platforms but I guess if we warn users about that issue they can be alert to danger.

I would like a statement from the developers that the number will always keep growing, do you want to ask them or should I ?

Davo

monperrus commented 4 years ago

I have built a nextcloud install (wow, finally found a use for Snaps!) and have some more data. Interesting.

cool :)

Note IDs are not allocated sequentially, indeed they jump a lot more than one per note, Not unique across platforms but I guess if we warn users about that issue they can be alert to danger.

Agree, this is good enough IMHO

I would like a statement from the developers that the number will always keep growing

I've browsed the Nextcloud code.

AFAIU, the id comes from method getId from interface Node with the implementation of FileInfo.

This id is actually a database field called file_id, confirmed by https://github.com/nextcloud/notes/blob/017e825dafd4212c362197f7def97bf225f17997/lib/Db/MetaMapper.php#L42

It seems that this database field is created by $fileId = $builder->getLastInsertId(); see https://github.com/nextcloud/server/blob/67ecdc969cd93442918180e179e1f05ed8618d92/lib/private/Files/Cache/Cache.php#L295

Per my understanding of the database code, this is is unique and most likely increases monotonically.

Good news!

davidbannon commented 4 years ago

Thought you might like an update. I have not dropped the ball.... I have working now a rough mockup of a tomboy-ng to NextCloud sync. While its working and I think, reasonably satisfactory, it does have some limitations that end users will need to work with. For the record -

I have not yet addressed any SSL issues. Nor keeping the NextCloud password in a safe place, thats looking like a platform by platform problem, sigh ....

I am also a bit concerned by the extra bulk this is adding to tomboy-ng, there is quite a lot of extra code required, particularly for the TomboyXML to markup conversion. I really don't know how many users will find this a useful addition. One approach may be to make a separate, standalone app that does the sync, only the tomboy-ng AND NextCloud users would need it. It would have to talk to tomboy-ng when sync-ing, else we'd risk changing a note just as it was synced.

Hmm....

Davo

monperrus commented 4 years ago

I have working now a rough mockup of a tomboy-ng to NextCloud sync

Oh great! I volunteer to beta-test it!

I am also a bit concerned by the extra bulk this is adding to tomboy-ng,

I think it's a great idea to keep it here, in order to have easy deployment and dissemination in distros having tomboy-ng, esp. Debian

davidbannon commented 4 years ago

OK, I have a working system I think. However, I have not tested it against an SSL Nextcloud server. Seems I cannot get SSL onto my Multipass Nextcloud (because it does not have a proper FQHN and the commercial provider I signed up for a free account with, QLoad (chosen at random), does not have the notes application installed and does not let me install a new application of my choosing. Is that normal ?

So, can you recommend a more suited provider ? I'm reluctant to go for a paid one, have no need for it. Or do you have a system available to try out against ? If so, let me know what OS you want the tomboy-ng client to run on (and assure me you understand about backups).

David

sliwowitz commented 4 years ago

Or do you have a system available to try out against ?

I have my own SSL server, I can set up a nextcloud account there for you to test if you want.

davidbannon commented 4 years ago

Sure, that would be great, thank you ! I would need only Notes, a few hundred megs, no more than a gig and that only for a short time. User=dbannon if possible. Davo

sliwowitz commented 4 years ago

@davidbannon OK. I've sent you an email with the login.

davidbannon commented 4 years ago

OK, I have placed three debs in the tomboy-ng releases page, a 32bit gtk version and a 64bit in GTK2 and Qt5.

The files are at https://github.com/tomboy-notes/tomboy-ng/releases/tag/v0.30 but are not mentioned on the Wiki release page due to their untested nature.

They are debs because its easy and what I use, if someone wants an RPM (or a general purpose tarball) please ask, almost as easy. Not too hard to do a windows install kit but a MacOS might be a bit awkward managing the ssl side.

Make sure you have good backups and read the note that the initial sync will create on your system (at both ends).

Please report your findings.

Thanks to sliwowitz for access to a nicely managed NextCloud server for testing.

Davo

monperrus commented 4 years ago

Nice, I'll try and let you know.

monperrus commented 4 years ago

Just installed https://github.com/tomboy-notes/tomboy-ng/releases/download/v0.30/tomboy-ng_0.30-0_amd64.deb

In the Seetings >> Sync tab, I don't see anything related to Nextcloud. Any idea?

davidbannon commented 4 years ago

Hmm, see the screen dump of the sync setting page ? Yours does not look like that ? I reckon you must have an earlier version somehow on your path. Try the 'About' menu item and see if you are running v0.30a. Sync-NextCloud

monperrus commented 4 years ago

Indeed I installed 0.30. Now installed 0.30a, https://github.com/tomboy-notes/tomboy-ng/releases/download/v0.30/tomboy-ng_0.30a-0_amd64.deb

monperrus commented 4 years ago

I can see the Nextcloud configuration screen!

monperrus commented 4 years ago

It works!

First feedback: if the password field contain spaces or newlines at the beginning or at the end, it would be great to trim them.

davidbannon commented 4 years ago

First feedback: if the password field contain spaces or newlines at the beginning or at the end, it would be great to trim them.

I am not sure about that. On some systems, spaces in passwords are permitted, but probably not at beginning or end. Easy enough to do but I wonder how an errant space has got in there in the first place. I guess we could have a 'show password' option.

monperrus commented 4 years ago

When I first connected, I had an error "authentication failed", it took me 5 minutes to realize I copy-pasted the password with a newline.

davidbannon commented 4 years ago

Yeah, that would do it ! And most people will be copy and pasting credentials from some sort of password manager I guess ....

So, strip any leading or trailing white spaces or newline chars ? Probably ...

Davo

monperrus commented 4 years ago

Here are the results of my first tests, that's really cool

What works:

What does not work:

Thanks Davo!

davidbannon commented 4 years ago

Interesting ! I was aware of the first issue (but forgot to follow up on it ). I suspect that when you rename a note in NextCloud, its last change date is not updated, therefore the sync engine does not know it needs to be updated. But I have not yet confirmed that. Will need to.

The second one has me a little unsure what you mean. You "create a note on a server, sync, see it on desktop" - "the note does not appear on desktop". Er, if you see it on the desktop, it must have appeared on the desktop ? Can you please clarify ? You make a new note using NextCloud, sync, and you don't then see it in tomboy-ng ? Thats something I thought I had tested pretty thoroughly but maybe I missed something. Was, I must admit, done in something of a rush....

monperrus, I will try and chase the first problem down, and reproduce the second one if I can but I will be pushed for time over the next few weeks. So, may take a bit of time.

Davo

monperrus commented 4 years ago

Hi Davo,

"create a note on server, sync, see it on desktop " is the test scenario. "the note does not appear on desktop" is the test result.

--Martin

davidbannon commented 4 years ago

OK, finally got a bit of time !

Davo

davidbannon commented 4 years ago

Item one above - https://github.com/nextcloud/notes/issues/621

davidbannon commented 4 years ago

Item three above - I cannot replicate those GTK warnings nor the memory leak. 24 bytes, too small to be something I allocated, have to assume its some GTK thing, will watch for it but assume its not an issue.

Davo

monperrus commented 4 years ago

Thanks for the update. Tried again test create a note on server, sync, see it on desktop, it works. Updated my test results above accordingly

davidbannon commented 4 years ago

Thanks for testing. It would be nicer if we could find a reason why it failed the first time. I don't like problems that just 'go away'. They usually come back !

There is/was a problem where a note would go wrong if it was in a category that included, in its name, a html illegal character, specifically '&', '<' and '>'. Fixed here so be careful until next beta releases....

Davo

davidbannon commented 4 years ago

I can now replicate both the GTK warnings and the memory leak. To see either you have to compile with GTK (slaps forehead). The GTK warning are a bug in Lazarus Trunk, I use trunk for dev work but not release. Have reported it and its been fixed already. The memory leak is more mystifying. Its not repeatable, seems random. Never present in Qt5 version so I still suspect its a GTK issue. But until I can make it repeatable, no hope of getting anything I can fix or report.