Prevent reuploading deleted files from Mobile App

xeonixo commented 8 months ago

The bug

I currently have the following problem - me and my girlfriend synchronize to the same account from our iPhones. When I try to delete the immich photos because of duplicates, the pictures are always synchronized again. Is there any way to prevent the immich app on the smartphone from trying to upload everything in the first place? I imagine it to be like a "fobidden list" for SHA's or unique IDs. The Duplicate-Detection wont work because of different file names or types for the same image.

The OS that Immich Server is running on

Debain 12

Version of Immich Server

v1.99.0

Version of Immich Mobile App

v1.99.0

Platform with the issue

[X] Server
[ ] Web
[X] Mobile

Your docker-compose.yml content

version: '3.8'
name: immich
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    networks:
      caddy:
        ipv4_address: 172.18.0.10
  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    #   file: hwaccel.transcoding.yml
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    networks:
      caddy:
        ipv4_address: 172.18.0.11

  immich-machine-learning:
    container_name: immich_machine_learning
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    #   file: hwaccel.ml.yml
    #   service: cpu
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    networks:
      caddy:
        ipv4_address: 172.18.0.12
  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    restart: always
    networks:
      caddy:
        ipv4_address: 172.18.0.15
  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
    networks:
      caddy:
        ipv4_address: 172.18.0.13
  immich-duplicate-browser:
    container_name: immich-duplicate-browser
    image: ghcr.io/agross/immich-duplicates-browser:latest
    environment:
      - IMMICH_URL=https://(redacted)
    networks:
      caddy:
        ipv4_address: 172.18.0.14
networks:
  caddy:
    external: true
volumes:
  pgdata:
  model-cache:

Your .env content

UPLOAD_LOCATION=./library
IMMICH_VERSION=release
DB_PASSWORD=(redacted)
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Sync with Mobile App
2. Delete File from immich - remove from trash
3. Sync again with Mobile App
4. Files are Back inside the immich

Additional information

No response

JW-CH commented 8 months ago

Delete File from immich - remove from trash

I assume you delete them in web, not on mobile right?

xeonixo commented 8 months ago

I assume you delete them in web, not on mobile right?

Yes, I am deleting in Web or via API.

Does it make a difference to delete inside the app?

Mostly I want to get rid of screenshots and some useless shots from work wich are needed on the phone but are useless on my Immich instance.

alextran1502 commented 8 months ago

We have yet to build a mechanism to not letting you reupload files that have been removed from the web yet

xeonixo commented 8 months ago

We have yet to build a mechanism to not letting you reupload files that have been removed from the web yet

Is that on any agenda for the next time?

alextran1502 commented 8 months ago

We have yet to build a mechanism to not letting you reupload files that have been removed from the web yet

Is that on any agenda for the next time?

No there is not

davidwaze commented 8 months ago

Hello,

I must admit this is very annoying for me too with same expectations.

There should be an option into the smartphone app as a first step to upload pictures once only (instead of having a full picture id's database). It shouldn't be too difficult right ?

When I want to do a picture cleanup into the common library and uploaded by somebody else than me. I need to identify which picture has been uploaded by who (we are using the same account to keep face recognition and so on). This is driving me insane 🤯.

My only workaround, but I don't like it, is I'm currently having a tasker scenario for my Android smartphones removing pictures older than 1 week. So I know when I deleting from my smartphone or through the web app something older thant 1 week, pictures won't be uploaded again accidentally.

Thanks again for your support.

krzemyczek commented 4 months ago

i have similar issue when i delete duplicate photos in Utilities and then my phone re-sync all of them :) a little annoying to be honest

harababurel commented 3 months ago

Is there any way to prioritize this?

It completely breaks a couple of use cases:

Cleaning up photos on the web UI (many of them will just get automatically re-uploaded from mobile)
- As such, the only workaround I can find is to clean up photos manually in the iOS Photos app, then do the same in Immich and make sure to delete the same set of photos. This is very tedious and error prone.
Duplicate detection, like @xeonixo mentioned above (Immich will delete duplicates but they also just get re-uploaded because they are still present on my phone).

Having an automated "blocklist" could help: when a photo is deleted on Immich, mark its asset ID somewhere and implement a mechanism to reject further automatic uploads of the same asset. This can naturally be configurable so that users can decide if they want to enable it.

Fixing this would improve the user experience a lot. Right now I've resigned myself to using Immich as a "dump everything in, don't bother organizing" sort of library.

Thanks!

C-Otto commented 3 months ago

As far as I understand the mobile code, all relevant state is provided by the server. When (not) uploading assets to the server, the mobile client first retrieves a list of all assets known for that device. Only assets that are not already known on the server are considered for an upload. The issue here is that the server "forgets" about this asset-device relationship once the asset itself is deleted (only) from the server (assets currently in the trash do not have this issue, though).

The mobile client, however, is already able to handle duplicates, so that these are not uploaded again once the server rejects the upload because of it being a duplicate. A very similar logic could be used to store the IDs/hashes/... of successfully uploaded files, so that these can (configurably?) be ignored from any future upload attempt.

Related code:

backup.service.dart: _saveDuplicatedAssetIds
backup.provider.art: _updateBackupAssetCount

C-Otto commented 3 months ago

Another approach could be to let the server remember which assets have been deleted (device-specific) so that said device can follow-up once it synchronizes. This way the pictures would also disappear from the mobile device, which I think is great - but that's debatable I guess. Do you know of an existing discussion of this feature?

AFumi39 commented 2 months ago

This is very annoying... I spent days to deduplicate thousands of photos from the Web UI. Tonight I cleared trash and then I opened the Mobile App (Android) and discovered that it started to upload again few thousands of photos. I just want to cry 'cause I know that I have to do ita again from the start... And it will be useless, because the mobile app will upload it again and again :(

I think that a solution can be a flag in the mobile app that allow to upload images "from now" (only new added images, like Synology do), or a sort of "blacklist" on the server of the deleted images (but that can be managed in some way, for example if you want to force the reupload of a specifica image)

C-Otto commented 2 months ago

If you have a database backup from in between, you might want to use that instead.

AFumi39 commented 2 months ago

If you have a database backup from in between, you might want to use that instead.

Unfortunately, I don't have a backup after the deduplication, I have at most the backup before the last ~2000 duplicated resolution. That can't be a solution, anyway, because after 30 days the deleted items are automatically removed, so the problem will rise again. I need to find a valid alternative before trying the deduplication again :(

C-Otto commented 2 months ago

You can just wait until this issue is fixed, you should get a notification.

AFumi39 commented 2 months ago

You can just wait until this issue is fixed, you should get a notification.

Sure, I will wait the fix before trying again ^_^

Thank you

AFumi39 commented 2 months ago

I'd like to add a use-case that I think it's very common. If you "migrate" from Google Photo to Immich, the first thing you'd probably do is the migration of the existing photos using takeout. Or, in any case, it's a thing you'll do sooner or later. Google Photo compress (by default) the images, so any single image you get from GP or Takeout is different from the ones you have on your device. This will cause the duplication of ALL the images you have on your smartphone, 'cause you can only choose if the App will upload anything or nothing. Synology Photo Android app have a useful flag that allow to upload only newer images that you take from that moment onward (I don't know if they use the "DateCreated" or if the app uses a DB to track all the old images not to upload)... And it works well!

So I think that a similar feature can fix many "duplication issues". A mechanism to prevent reupload of the same image that you delete on the server, is needed too, but I think that if the "only newer" feature is implemented, it can work as workaround of many that duplication cases (to take time).

Thank you

Bananas-Are-Yellow commented 1 day ago

Same problem is described in #12051.

immich-app / immich