Auto-upload feature sometimes fails to upload file correctly - results into incomplete/broken file in the cloud.

scr4bble commented 11 months ago

⚠️ Before posting ⚠️

[X] This is a bug, not a question or an enhancement.
[X] I've searched for similar issues and didn't find a duplicate.
[X] I've written a clear and descriptive title for this issue, not just "Bug" or "Crash".
[X] I agree to follow Nextcloud's Code of Conduct.

Steps to reproduce

have autoupload active for folder where Camera stores new photos/videos
record new video
wait for the nextcloud to start uploading
cause some network issue and hope to reproduce the failure that results into incomplete file being uploaded

Expected behaviour

File should either be uploaded completely or not at all = upload postponed and/or repeated later, trying until the file is successfully uploaded.

Actual behaviour

My video file has ~ 150 MB locally in my android phone but the file uploaded via auto-upload feature has 40 MB and I am unable to play/open the cloud version of the file. The file uploaded into cloud is broken and incomplete.

Android version

12 SKQ1.211019.001

Device brand and model

Xiaomi Redmi Note 9 Pro

Stock or custom OS?

Stock

Nextcloud android app version

3.26.0

Nextcloud server version

27.1.5

Using a reverse proxy?

I don't know

Android logs

No response

Server error logs

No response

Additional information

I am not hosting the server myself and I couldn't fine the android nextcloud logs either: (looking for nextcloud.log - nothing was found on the device).

scr4bble commented 11 months ago

I just found out that there are more incomplete video files in my nextcloud instance. Seems like this is happening quite often (1/10 video files are incomplete). I am able to play the uploaded part of the video file via web-ui - the player writes "Error loading XYZ.mp4" when it reaches the breaking point in the file. On mobile I am unable to play such incomplete video file at all.

I would expect the client application to employ some hash checking to make sure the file was uploaded correctly. I really didn't expect this to happen. Backup is one of the main reasons for having nextcloud. It has to be reliable, otherwise it makes no sense to use it.

scr4bble commented 11 months ago

We need solution to this issue that also fixes all existing files that were not uploaded correctly...

joshtrichards commented 11 months ago

Hi @scr4bble -

Were there any notifications about upload failures?
Are any of these files listed in your Uploads tab? What is their state there?
Can you also share your auto-upload settings for this folder?

scr4bble commented 11 months ago

Hi @joshtrichards,

I remember some error notification from at least one of these cases. I think I just swiped them away in expectation that it simply attempts a reupload later on.
Yes, I just checked one of these incorrectly uploaded videos (the latest one) and I see the file listed in "Uploaded" section. It is not listed in "Failed/pending restart" section

Auto-upload settings

Checked boxes:
- Only upload on unmetered Wi-Fi
- Also upload existing files
(other checkboxes are unchecked)
Original file will be... "kept in original folder"
What to do if the file already exists? "Skip uploading"

joshtrichards commented 11 months ago

I am not hosting the server myself and I couldn't fine the android nextcloud logs either: (looking for nextcloud.log - nothing was found on the device).

Unfortunately if you're not managing the server yourself you may not have access to the logs there, but the administrator would. It would be interesting to see what the web server saw in terms of HTTP status code (for every transaction involving one of the incomplete files), to confirm this is solely client matter and not some server interaction.

What would be most helpful though you can probably do: Gathering the debug logs from the app while it's running on your phone and triggering one of these faulty uploads. Unless you have a rooted phone, this will require using adb[1]. That's one approach.

Another approach, if this is relatively easy to reproduce on your end (hopefully), would be to try to do so using a Dev edition of the app. The Dev edition includes built-in log access under Settings->Logs. You can install the Dev edition in parallel with your existing Stable edition app. Just point the Dev edition at a different auto-upload source folder (i.e. A test one) / don't configure any overlapping auto-upload folders between the two. The Dev edition can be most easily installed via F-Droid[2].

[1] https://github.com/nextcloud/android?tab=readme-ov-file#getting-debug-info-via-logcat-mag [2] https://f-droid.org/en/packages/com.nextcloud.android.beta/

andreclemente commented 8 months ago

I also have this issue without causing any network issue. I have multiple photos and videos corrupted in nextcloud that are now removed from the phone (automatically) and now I don't have them anywhere...

scr4bble commented 7 months ago

Yes, this is an unacceptable behavior for any backup software. I am already forced to look for a replacement. I was hoping FOSS world is able to provide a good and reliable solution as an alternative to the big cloud providers but it doesn't seem to be a case as this is by far not the only issue of NextCloud.

Few points:

Reliability in backup is A MUST.
It cannot flag an upload to be finished if there was an error.
Sending error message as notification on the client IS NOT ENOUGH - software has to repeat the failed upload until it is successful
After each upload it has to apply some hashing alghoritm to VERIFY the file was uploaded successfully (compare file on the server and on the client).
as @andreclemente pointed out it is unacceptable for the client app to erase a file that wasn't uploaded successfully

If you don't do this properly, then RIP NextCloud - people will never adopt it on bigger scale. You have to change something drastically and really fast.

nextcloud / android