Sync API timing problems

[davidbannon]

This relates to #621 but is much more specific and probably better presented as a stand alone issue. I am building a sync between tomboy-ng and Nextcloud Notes. The Tomboy standard relies on meta data stored in xml and I am finding difficulties working with NextCloud which does not.

Unless there is a way around this issue (that I cannot see) it is probably also causing problems with other applications using the API.

Because the 'modified' time returned via the API does not respect things like changes to a note's name, I must use the pruneBefore parameter to a GET to determine what notes have changed on NextCloud. So, the process is to request a list of notes from NextCloud and record its time stamp (LastSyncTime). We work out what needs to happen and download and upload as necessary. That takes some time so we end up with uploaded notes getting NextCloud time stamps later than the recorded LastSyncTime.

If no changes are made at either end and another sync is run, the uploaded notes (remember they were uploaded after the LastSyncTime) are seen as new or changed and will be downloaded. Putting aside the network traffic issue, they will then get a wrong LastChangeDate and will trigger a false 'clash' if the user does make a change locally between those syncs. Confusion and potential data loss.

A similar problem happens if new/changed notes are uploaded before taking a pruneBefore GET.

Is there a way to avoid this that I have not spotted ? It seems to me that the difficulties will be with us unless NextCloud Notes finds a way to maintain the necessary meta data outside of the note files them selves.

NextCloud seems to be maintaining two time stamps, the one associated with 'modified' and the one that is used by pruneBefore. I'm guessing that pruneBefore uses the file's file system date stamp, just where does 'modified' time come from ? Personally, I believe that 'modified' should mean any modification including the note name !

Davo

[davidbannon]

Another approach would be to allow a means to set the (pruneBefore) time on a note when its uploaded. Settings it's 'modified' time does not help by the way.

If I am right that pruneBefore uses the file system datetime for the note file itself, then allowing it to be set would be hard but not impossible (given that I have no idea what interaction is allowed by Nextcloud to filesystem).

It would be nice to see a bit of documentation about these two time stamps and how they are used.

Davo

[korelstar]

Hey @davidbannon, I'm very sorry for my late reply. Unfortunately, I am very busy since some months.

I understand that a real last-modified time (which includes all metadata) would be nice, but unfortunately the Nextcloud architecture doesn't provide such a time and, however, I don't think it is necessary. Let's have a look behind the scenes:

In Nextcloud, every file has a filemtime, i.e. the modified time of the file. It is important to know, that this time can be updated separately from updating content (like in every filesystem). Hence, it is possible that the filemtime doesn't change even if the file content does change. Furthermore, the filemtime is updated only if the content of the file is changed. If you rename a file or add a file to your favorites, the filemtime does not change (like in every filesystem).

In the notes app, we use this filemtime as modified attribute for notes. This attribute indicates the last change of a note's content on an application-level (not on an technical level!). This is important, because we don't want to set this time to the last sync time (e.g. if you change a note in Tomboy at 9:00 and sync that note at 10:00, the modified time on the server must be 9:00 and not 10:00 -- if you then move a note to another category at 11:00, the modified time must still be 9:00)

Hence, the modified time cannot be use for technical purposes like synchronization! In order to improve synchronization, we internally (!) use an attribute lastUpdate. Please see the comment in this file for further information about this: https://github.com/nextcloud/notes/blob/master/lib/Service/MetaService.php

It is important to understand, that this lastUpdate attribute is also not sufficient for your purposes, because it does not always contains the exact change time, but a time t that is at some point between the real changed time and the next synchronization time. This is totally sufficient for our (internal) purpose, so this time can be used for pruneBefore.

The problem is, that the Nextcloud architecture does not allows us to use an exact date, because we cannot track all changes made on the note's file and note's metadata. Therefore, other third-party apps using the notes API do not rely on times, but manage a local attribute to track whether there was a local change. Only if there was a local change, the note is uploaded to the server. On the other hand, all notes in the server response (using pruneBefore) are saved/updated to the local store. Couldn't this be a solution for you, too?

Otherwise we could think of providing lastUpdate using the API. Would that help? (be aware it is not exact!)

[davidbannon]

Sorry Korelstar, I have been quite busy recently. I did try and get a sync going using lastupdate but, as I modeled on paper, it was very easy to get things crossed over. I do really believe this problem exists in any sync app hitting this API. And sorry to have to say, it will drop user data from time to time.

Thanks for your help with this Korelstar, I'll put my plans on hold for now, maybe something will happen in the future ?

Davo

[korelstar]

The issue with lost updates is documented in #56 (see also https://github.com/nextcloud/notes/issues/621#issuecomment-716102458). If you see any problems that are going beyond this, please give a step-by-step example. Then, we can fix this.