[BUG] Getting many duplicated photos

[nsantini]

Describe the bug I connected the app to the server and backed up all the photos by enabling auto backup. Then I downloaded my Google Photos and uploaded them to the server via the CLI tool. I'm now seeing lots of "duplicates" both on the mobile app and on the web server.

Task List

Please complete the task list below. We need this information to help us reproduce the bug or point out problems in your setup. You are not providing enough info may delay our effort to help you.

• [ x ] I have read thoroughly the README setup and installation instructions.
• [ x ] I have included my docker-compose file.
• [ x ] I have included my redacted .env file.
• [ x ] I have included information on my machine, and environment.

To Reproduce Steps to reproduce the behavior:

1. Enable auto backup in the phone
2. Upload photos via the CLI tool

Expected behavior When the phone gets access to the CLI uploaded photos it does not duplicate them by re-uploading them

Screenshots On the mobile app the duplicates have two different icons, one with a cloud and a tick, the other with a photo icon.

System

• Phone OS: iOS: 16.0
• Server Version: v1.29.5
• Mobile App Version: 1.29.4 build.59

Docker compose

version: "3.8"

services:
  immich-server:
    image: altran1502/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    image: altran1502/immich-server:release
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    image: altran1502/immich-machine-learning:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - database
    restart: always

  immich-web:
    image: altran1502/immich-web:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    env_file:
   - .env
    environment:
      - PUBLIC_TZ=${TZ}
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: altran1502/immich-proxy:release
    ports:
      - 2283:8080
    logging:
      driver: none
    depends_on:
      - immich-server
    restart: always

volumes:
  pgdata:

Env File

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=<<redacted>>
DB_USERNAME=<<redacted>>
DB_PASSWORD=<<redacted>>
DB_DATABASE_NAME=<<redacted>>

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# Optional Redis settings:
# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Config
###################################################################################

UPLOAD_LOCATION=<<redacted>>

###################################################################################
# Log message level - [simple|verbose]
###################################################################################

LOG_LEVEL=simple

###################################################################################
# JWT SECRET
###################################################################################

JWT_SECRET=<<redacted>>

###################################################################################
# MAPBOX
####################################################################################

# ENABLE_MAPBOX is either true of false -> if true, you have to provide MAPBOX_KEY
ENABLE_MAPBOX=false
MAPBOX_KEY=

####################################################################################
# WEB - Optional
####################################################################################

# Custom message on the login page, should be written in HTML form.
# For example PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"

PUBLIC_LOGIN_PAGE_MESSAGE=

# For correctly display your local time zone on the web, you can set the time zone here.
# Should work fine by default value, however, in case of incorrect timezone in EXIF, this value
# should be set to the correct timezone.
# Command to get timezone:
# - Linux: curl -s http://ip-api.com/json/ | grep -oP '(?<=timezone":")(.*?)(?=")'

TZ=Pacific/Auckland

[zackpollard]

Hey! So I am currently doing the same and working through a bunch of issues this caused for me. Personally I am currently planning to just accept the inevitable duplicates. The reason this happens will be if you had "High Quality" mode enabled in google photos. Essentially this doesn't save the original file in google photos and instead, it compresses the file. Currently our duplicate detection is only for photos that are exactly the same, however, we are planning on building some tools later into the UI that can detect similar photos, which should help for these kind of cases where the photos are visually the same, but different on disk. I'm going to close this for now as it is expected behaviour, however we do plan to have a solution to this at some point, but it might not be until after our first major release.

[nsantini]

Thanks @zackpollard for the explanation. Do you have any idea of how to approach a manual dedup process?

Also, about those two different Icons I see on the duplicated photos on the mobile app. What do they mean? one on device and one on server? Would be great to understand that so I know how to deal with them

[nsantini]

Also, should I try to delete duplicates on the mobile app or on in the web view?

[nsantini]

Ok, I figured out. In the mobile app, the ones with the cloud are those in the server, which came from google photos. The ones with the pic icon are on the phone (higher quality). So. by deleting from the phone those in the server, I get rid of the duplicates and keep the higher quality versions

[zackpollard]

Ok, I figured out. In the mobile app, the ones with the cloud are those in the server, which came from google photos. The ones with the pic icon are on the phone (higher quality). So. by deleting from the phone those in the server, I get rid of the duplicates and keep the higher quality versions

Yes I believe that is correct. God speed in manually going through all your photos!