Files with same size but different timestamp/checksum are not synced

kaoneko commented 7 months ago

Edit: Read this comment for a TLDR; and analysis as of May 5th.

[✔️] I have read the above note

What version of Round Sync are you using (About -> App version)?

2.2.3-terrapin

What is your Android version, phone model and manufacturer?

Android 13, Google Pixel 4a

Which steps are required to reproduce this issue?

I noticed some daily backups were not updated on my remote. They are files with the same size but different contents and a different timestamp. Activating Use md5 checksum made no difference. I'll walk you through my setup, but I don't know if it's specific to a Crypt remote on a WebDAV remote. #37 implies this topic still needs investigation.

Create a WebDAV remote
Create a Crypt remote pointing to a folder on the WebDAV remote
Create a local folder with a text file with contents abc
Create the same folder and file on the Crypt remote but make the contents of the file 123
Check that both files have the same size but different timestamps
Create a task that syncs the local folder to the remote folder ("Sync local to remote", with or without "Use md5 checksum")
Run the task, the result is "There was nothing to do" and the local file is not copied to the remote

Please also enable rclone logging (Settings > Logging > Log Rclone errors). You're going to need the log for the last question.

No log file was written after enabling this setting and running the task. After force closing and restarting Round Sync and running the task, a log file was written though. It's one of the bugs I alluded to in my recent report.

Despite your statement, the log file was not needed to answer the last question.

What is your configuration (`rclone.conf`)?

Note: these are not my real credentials and webdav host.

[Redacted]
type = webdav
rclone_remote_name = Redacted
pass = hunter2
user = AzureDiamond
url = https://webdav.redacted.com

[backup]
type = crypt
password = hunter2hunter2
directory_name_encryption = true
remote = Redacted:backup

[3d11ef8a-387b-3205-e52f-26e3c7e172e6]
type = alias
remote = /storage/emulated/0

Does the same issue also occur when using the same configuration on a PC or in Termux?

Tested on PC with rclone sync /local/folder/ backup:folder/: local file was copied to remote Tested on Android with S3Drive: local file was copied to remote

kaoneko commented 7 months ago

Alright, I've looked into this some more and found out a few things:

Taking the crypt out of the equation doesn't influence the bug
The issue doesn't occur with a Koofr remote
On the Koofr remote I can tap to calculate the MD5 or SHA1 checksum of a file in its Properties popup, the first returns a checksum, the second UNSUPPORTED; on the WebDAV remote both return ERROR

Now the rclone documentation says:

Plain WebDAV does not support modified times. However when used with Fastmail Files, Owncloud or Nextcloud rclone will support modified times.

Likewise plain WebDAV does not support hashes, however when used with Fastmail Files, Owncloud or Nextcloud rclone will support SHA1 and MD5 hashes.

So now it makes sense right? I guess this bug is not solvable for plain WebDAV remotes then?

However, I'm using Fastmail. But as you can see in the exported configuration in my opening post, the [Redacted] entry is missing the line vendor = fastmail. This remote was configured using Round Sync, including selecting the vendor type (which, like some other fields, also doesn't show a value when you edit the remote again, by the way).

The rclone.conf on my PC does have the vendor line. So I added it to the config file in the Round Sync export and imported it. And what do you know? Syncing a file with the same size but different timestamp/checksum on local vs remote now works when Use md5 checksum is enabled in the task! Yay.

At the same time, the Tap to calculate MD5 or SHA1 still both return an error for most files I tested it with. I've got one file which does show a SHA1 checksum somehow...

I also encountered yet another bug when figuring out this one: when importing a Round Sync backup which has md5sum: true in the JSON of every task in rcx.json, all imported tasks will actually have Use md5 checksum disabled, and when you then export a new config backup, this is also reflected in the JSON.

At this point I am somewhat shocked at the sheer amount of severe bugs (bugs that may lead to data loss) I've already encountered in Round Sync, i.e. this one, using the checksum being disabled on import, scheduled Triggers not running, a new remote overwriting an existing remote without warning when it has the same name. Add to that the many other weird bugs and I'm not sure if I can trust this software with my data.

Is this something you guys are aware of? I see a new "stable" version has just been released, I haven't checked it out yet. Is it a step forward?

The beta releases have a clear warning on them. I think the 2.2.3 version should've also had that warning and the beta label, because it's not stable at all in my book.

I really think this is app has potential, and there's already so many things it does right as far as I can see, like the clean interface, being able to configure remotes in the app, scheduling (if it can be trusted) and the content provider! So I'm not here to criticize it, but to help it get better, because I really want it to be better!

Love to hear your thoughts, devs & contributors ❤️

newhinton commented 7 months ago

Thanks for looking into those issues, that is very helpful to me.

Let me first adress the comment regarding existing bugs and beeing aware of them:

I am more or less the sole developer of this app, and i am certainly not aware of many bugs. My time is very limited, and there is a lot of old code and required maintenance, so writing tests hasn't come up. I am relying on people like you to report specific issues, that i then can adress.

Now to the rest:

If i understand, this is three seperate issues:

Remote creation overwriting existing remotes
Remote editing not listing formerly changed values (could you list the fields you did find that were not properly repopulating?)
Import/Export not properly respecting md5sum

The other issues are already seperate tickets. Does this properly summarize your findings?

mtalexan commented 6 months ago

It sounded to me like the root of the specific problem was the vendor not being set in the actual rclone config file for the WebDav type when it was set in the Round Sync GUI.

Also it sounds like maybe the Calculate checksum option on files viewed from the built in remote browser(?) errors out rather than calculating the checksum if the endpoint type doesn't support timestamps (?).

kaoneko commented 6 months ago

@newhinton

I am more or less the sole developer of this app, and i am certainly not aware of many bugs. My time is very limited, and there is a lot of old code and required maintenance, so writing tests hasn't come up. I am relying on people like you to report specific issues, that i then can adress.

Fair enough, I'll see if I can add some solid bug reports on the issues I encountered. Thank you for your work on Round Sync 🙏🏻

If i understand, this is three seperate issues:

Remote creation overwriting existing remotes

Correct. I'll test if this still happens in the latest version and open an issue about this.

Remote editing not listing formerly changed values (could you list the fields you did find that were not properly repopulating?)

This doesn't seem to be an issue anymore in the current release 👍🏻

Import/Export not properly respecting md5sum

Correct. I'll do a proper bug report in a separate issue.

The other issues are already seperate tickets. Does this properly summarize your findings?

Yes.

@mtalexan

It sounded to me like the root of the specific problem was the vendor not being set in the actual rclone config file for the WebDav type when it was set in the Round Sync GUI.

Yes, this must be causing the issue for WebDAV remotes with a vendor that supports hashes and modification times.

Also it sounds like maybe the Calculate checksum option on files viewed from the built in remote browser(?) errors out rather than calculating the checksum if the endpoint type doesn't support timestamps (?).

Yes, Calculate checksum was used from the built-in remote browser. It errors out even after importing an rclone.conf with vendor = fastmail, apart from the SHA1 checksum on one particular (and particularly large [440 MB]) file...

Not solved for crypt remote

In the meantime I found out that, after adding the vendor = fastmail line to Round Sync, the syncing a newer file with the same size issue is NOT solved when syncing to the crypt hosted on my WebDAV remote. As stated before, this issue does not occur with rclone on my laptop (which also uses rclone 1.66) nor with another rclone client for Android.

Possibly relevant pieces of documentation on https://rclone.org/crypt/:

Modification times and hashes

Crypt stores modification times using the underlying remote so support depends on that.

Hashes are not stored for crypt. However the data integrity is protected by an extremely strong crypto authenticator.

Use the rclone cryptcheck command to check the integrity of an encrypted remote instead of rclone check which can't check the checksums properly.

and

Rclone does not encrypt

file length - this can be calculated within 16 bytes

modification time - used for syncing

kaoneko commented 6 months ago

I'm not good with walls of text, so here's a recap for clarity.

Issue: Local file with same size as remote file but newer modification time and different hash/content is not uploaded to remote when syncing local to remote.

	Use md5 checksum	WebDAV	crypt-on-WebDAV
rclone.conf generated through app	off	issue	issue
	on	issue	issue
rclone.conf with `vendor = fastmail`	off	issue	issue
	on	solved	issue

Todo:

[ ] fix vendor type not added to rclone.conf
[ ] find out why issue still occurs on crypt

Related issue: Calculating checksums in file browser displays ERROR on WebDAV remote (for most files); not fixed by vendor = fastmail.

ndesgranges commented 1 month ago

Likewise plain WebDAV does not support hashes, however when used with Fastmail Files, Owncloud or Nextcloud rclone will support SHA1 and MD5 hashes.

I'm using plain webdav and this statement is not to my liking 😄

Why is rclone / round-Sync relying on server side checksum ? It might be very slightly slower to do everything locally, but this way most of the issues above should be fixed right ? (just wondering)

kaoneko commented 1 month ago

How much slower it would be depends entirely on the amount of data that needs to be synchronized and the speed of one's connection, since you need to download EVERYTHING to generate checksums locally for the remote files. This is not ideal, especially with frequent synchronization and/or a lot of data. So I understand the preference for letting the remote host calculate the hashes.

On the other hand, it depends entirely on the specific circumstances. If downloading your data from the remote only takes a few seconds, or you don't care how long it takes, and you have enough disk space or memory to temporarily store the files locally, calculating the hashes locally would indeed solve the issue of a newer file with the same size not being synced, without being impractical.

ndesgranges commented 1 month ago

Good point. I guess for little files, it becomes very CPU expensive (lot of checksums) And for large files storage/connexion expensive.

But anyways, this means it is not a solution even for plain Webdav, especially for large files.

Sadly, I'll have to go back to folderSync for my sync needs for now. In the meantime, I'll dive into rclone a little to see why this choice has been made. If I end up opening an issue, I'll link it here.

kaoneko commented 1 month ago

I was going to ask: if plain WebDAV does not support modified times or hashes, how is FolderSync solving this issue? But it seems I misinterpret the statement from the rclone documentation ever since I first read it.

I thought it meant files stored on a plain WebDAV server don't have any timestamp whatsoever, but after some reading I've come to the conclusion it just means when uploading a file, its last modified time is changed to the time of upload.

Which is still enough to know if the uploaded file is older than the local file in any scenario that only involves one device syncing to the WebDAV server. I guess FolderSync just relies on this (and risks data loss when multiple devices are involved)?

If files are new to FolderSync or has changed since last sync, they are compared using their last modification date and optionally using MD5 hashes (for local files and remote providers that offers this information).

— https://foldersync.io/docs/help/syncing/#file-comparison

Or does it also offer an option for local checksum calculation of the remote files? Or does it just compare file sizes on plain WebDAV servers instead and suffer from the same behavior this issue is about?

I know I didn't have this issue when I was using FolderSync (or I just never noticed it?), but then again, I'm using a Fastmail server which properly supports modified times (I don't know if FolderSync makes use of the Fastmail-specific implementation though) and my issue is just with Round Sync not getting this right.

ndesgranges commented 1 month ago

I guess FolderSync just relies on this (and risks data loss when multiple devices are involved)?

FolderSync offers the possibility to use checksum even on plain Webdav. The setting exists, but I do not know if it is actually used under the hood in the case of plain Webdav.

About conflicts, FolderSync does not bother and throw me an error in my case. I have the choice to define if the Host or the Client should have the last word. In my scenario where I only sync in one direction, this is not much of an issue, but I can imagine it could be frustrating in bidirectional scénarios.

newhinton / Round-Sync