Externally deleted files uploaded again (because external app isn't really deleting them)

simon-eller commented 2 years ago

⚠️ Before posting ⚠️

[X] This is a bug, not a question or an enhancement.
[X] I've searched for similar issues and didn't find a duplicate.
[X] I've written a clear and descriptive title for this issue, not just "Bug" or "Crash".
[X] I agree to follow Nextcloud's Code of Conduct.

Steps to reproduce

Configure auto upload in the Nextcloud app for the camera folder of the smartphone Shoot a photo and wait until it's uploaded on the Nextcoud server Delete the photo in the Samsung gallery app Go to your Nextloud app and click on "show hidden files" option Now you can see that the photo was uploaded twice

Expected behaviour

If the photo was deleted on the smartphone, nothing should actually happen with the current options on the Nextcloud itself.

Whether the photo should then also be deleted on the Nextcloud has already been discussed in a number of other issues.

Actual behaviour

If you delete a photo in Samsung Gallery, it will not be deleted directly. Instead, the file is renamed and only really deleted after 30 days (because of the recycle bin function).

Original file name: 20220717_1234.jpg Deleted file name: .trashed--20220717_1234.jpg

The deleted file will no longer appear in Samsung Gallery, but it is still there. The problem now is that Nextcloud thinks it is a new file and uploads this file as well.

The photo is now available twice on Nextcloud, except that the deleted version is not displayed directly because the file is classified as a hidden file by the dot at the beginning of the name.

The second version is only displayed as soon as you select the "Show hidden files" option. original_file_and_deleted_file

Android version

12

Device brand and model

Samsung Galaxy A51 SM-A515F/DSN

Stock or custom OS?

Stock

Nextcloud android app version

3.21.0

Nextcloud server version

24.0.3

Using a reverse proxy?

No

Android logs

No response

Server error logs

No response

Additional information

The Nextcloud app should recognize if it is a file with the name something like .trashed<...>.jpg and not synchronize it to the server.

AlvaroBrey commented 2 years ago

This should be treated as an enhancement, I think. We're not responsible for the behaviour of a particular photos app. We could try to detect this but who is to say that the next version of the Samsung Gallery will not rename them differently?

The only thing that would make sense to me here would be to add a setting to the autoupload folder to avoid uploading hidden files (possibly on by default).

t3pfaffe commented 1 year ago

This should be treated as an enhancement, I think. We're not responsible for the behaviour of a particular photos app.

@AlvaroBrey

This is the trash behavior of Google's default "Files" app as well on my Pixel 7. If it's the behavior of Android's default file browser as well as Samsung's, I'm pretty sure it can be treated as a standard Android behavior going forward. I agree it should be generalized to allow for ignoring custom file extensions, but you can't write this off as just a peculiarity of some niche photos app. This is just as much a bug as a feature enhancement.

joshtrichards commented 9 months ago

I presume you don't want to use the client's existing setting that enables delete the on-device photos after they're auto-uploaded?

pulsejet commented 4 months ago

I want to quickly bump this; this seems to be default behavior even when Photos are deleted using the native APIs or Google Photos. Is there any issue in simply excluding files with name matching ^.trashed-?

qugebert commented 3 months ago

Hi, I encountered this problem for the first time today. What surprises me a bit is that in the general settings, 'Show hidden files' is disabled, yet these files, which should be considered hidden, are still being uploaded. Would it make sense to generalize the problem to the extent of adding an option 'Ignore hidden files' (files starting with .) to the settings of Auto-Upload?

ofrias commented 3 months ago

Same problem here on a Google Pixel 6. The standard photos app renames deleted files as ".trashed..." and they are uploaded by Nextcloud app. This should not happen. Nextcloud app should not upload hidden files (those starting by a dot).

idle-user commented 2 months ago

There should be an "do not upload hidden files" or "ignore hidden files" flag. It's frustrating that they're being uploaded.

As others have mentioned, the standard method these file cleaners work is to mark the files for deletion by putting ".trashed-" prefixed to the files.

But the app reads these as new files and begins the upload process.

I also noticed some videos that are processing try to upload if the timing is right. They appear as ".pending-". This would solve that too

calando commented 2 months ago

I would second the request, but for a slightly different reason: in my setup, each generation of a .trashed_* file leads to a subsequent upload error (that means, I do not have duplicated photos on my NC instance, but I get lots of error notifications from the NC app on my phone).

No idea why the upload of these .trashed_* files does not work, though (NC instance is latest release version, runs behind a reverse proxy)

joshtrichards commented 2 months ago

Instead, the file is renamed and only really deleted after 30 days (because of the recycle bin function).

As a workaround, try using the option for your auto-upload profiles to only upload while charging.

ofrias commented 2 months ago

As a workaround, try using the option for your auto-upload profiles to only upload while charging.

In my case, I am already using the "only upload while charging" setting but this bug breaks my use case. I want to be able to delete the photos that I don't want to keep using Google Pixel default camera/gallery application. Then, when I plug the power cable, I want Nextcloud app to upload the new photos (but not the ones that I deleted). Since the Pixel renames to hidden files instead of deleting, and Nextcloud upload all hidden files, all photos end up being uploaded, even the ones deleted/renamed in the phone.

ghaberek commented 2 months ago

I was getting fed up with this so as a work-around, I've got Termux installed and wrote a Bash script to purge the .trashed files. In my case they're typically found in either DCIM/Camera or Pictures/Screenshots so those are the only folders I'm targeting.

#!/data/data/com.termux/files/usr/bin/bash
for folder in 'dcim/Camera' 'pictures/Screenshots'; do
  find $HOME/storage/$folder/ -type f -name '.trashed-*.jpg' -print -delete 2>/dev/null
done

(Note the storage folders use lower-case names in Termux, e.g. storage/dcim.)

Edit: I transposed this by hand and was missing the single quotes between the two folder names on line 2.

Aohzan commented 2 months ago

In addition, .pending files are uploaded too

zeugmatis commented 2 months ago

Bumping this, very aggravating when cleaning out galleries on the phone, it re-uploads useless dotfiles doubling space and also bandwidth usage. There should be an option to ignore dotfiles.

bioxz commented 1 month ago

I recently switched from a Pixel 4 to a Pixel 7 and now I get a lot of upload failures due to .pending files ("Local file not found") and . trashed files ("File could not be copied to local storage"), someone else above had the same issue. But in the end the outcome (failure or duplicate) does not matter. This is with the default photos applications on Android. While I do not think that the behavior of that photos app is "correct" (those files should go into cache and trash folders, not just given a different name in place), it's unlikely that someone here will be able to change that behavior.

Some kind of regex blacklist feature would be great to solve this and similar issues. Could be per upload folder or system wide. Alternatively just being able to avoid specific files (e.g. "ignore hidden files") would be sufficient here as well.

DecaTec commented 1 month ago

Same problem here.

As a workaround, you can use the "file access control" app and define a flow in order to deny access to these files: Filename matches /^\.trashed-.+$/i

This prevents uploads of these deleted photos, but it's still very ugly, because every time, you'll get an error in the Android app telling you that the access was denied while uploading these files.

At least these files do not end up being uploaded to your cloud...

Keep in mind to delete all .trashed-* files before activating this rule, otherwise you'll won't be able to delete this files afterwards in the cloud.

nextcloud / android