[BUG] Duplicate photos in timeline

[zsolt-donca]

The bug

Most of my photos and videos appear twice in the timeline, once with the cloud icon, and once again with the "cloud-off" icon (cloud with a slanted line). If I download one of the cloudy photos, the icon will have a check mark in it, as if Immich considered the two separate.

This happens on a clean install, with the mobile app uploading the first photos to the server.

Using a Pixel 7 with stock OS.

The OS that Immich Server is running on

Arch

Version of Immich Server

1.63.2

Version of Immich Mobile App

1.63.0 build.86

Platform with the issue

• [ ] Server
• [ ] Web
• [X] Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /var/local/immich/model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - /var/local/immich/tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - /var/local/immich/pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

Your .env content

###################################################################################
# Database
###################################################################################

# NOTE: The following four database variables support Docker secrets by adding a *_FILE suffix to the variable name
# See the docker-compose documentation on secrets for additional details: https://docs.docker.com/compose/compose-file/compose-file-v3/#secrets
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=<redacted>
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# REDIS_URL will be used to pass custom options to ioredis.
# Example for Sentinel
# {"sentinels":[{"host":"redis-sentinel-node-0","port":26379},{"host":"redis-sentinel-node-1","port":26379},{"host":"redis-sentinel-node-2","port":26379}],"name":"redis-sentinel"}
# REDIS_URL=ioredis://eyJzZW50aW5lbHMiOlt7Imhvc3QiOiJyZWRpcy1zZW50aW5lbDEiLCJwb3J0IjoyNjM3OX0seyJob3N0IjoicmVkaXMtc2VudGluZWwyIiwicG9ydCI6MjYzNzl9XSwibmFtZSI6Im15bWFzdGVyIn0=

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_USERNAME=
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/var/local/immich/store

###################################################################################
# Typesense
###################################################################################
TYPESENSE_API_KEY=<redacted>
# TYPESENSE_ENABLED=false
# TYPESENSE_URL uses base64 encoding for the nodes json.
# Example JSON that was used:
# [
#      { "host": "typesense-1.example.net", "port": "443", "protocol": "https" },
#      { "host": "typesense-2.example.net", "port": "443", "protocol": "https" },
#      { "host": "typesense-3.example.net", "port": "443", "protocol": "https" },
# ]
# TYPESENSE_URL=ha://WwogIHsgImhvc3QiOiAidHlwZXNlbnNlLTEuZXhhbXBsZS5uZXQiLCAicG9ydCI6ICI0NDMiLCAicHJvdG9jb2wiOiAiaHR0cHMiIH0sCiAgeyAiaG9zdCI6ICJ0eXBlc2Vuc2UtMi5leGFtcGxlLm5ldCIsICJwb3J0IjogIjQ0MyIsICJwcm90b2NvbCI6ICJodHRwcyIgfSwKICB7ICJob3N0IjogInR5cGVzZW5zZS0zLmV4YW1wbGUubmV0IiwgInBvcnQiOiAiNDQzIiwgInByb3RvY29sIjogImh0dHBzIiB9Cl0=

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

###################################################################################
# Immich Version - Optional
#
# This allows all immich docker images to be pinned to a specific version. By default,
# the version is "release" but could be a specific version, like "v1.59.0".
###################################################################################

#IMMICH_VERSION=

Reproduction steps

1. Install server
2. Create non-admin user
2. Enable public subdomain and HTTPS with letsencrypt certificate, behind nginx
3. Install mobile app
4. Login with non-admin user
5. Select some folders to backup
6. Observe the timeline showing duplicates

Additional information

No response

[zsolt-donca]

Screenshot:

[alextran1502]

Hello, if you logout and log back in, does it solve the issue?

[zsolt-donca]

I just tried that, signed out and signed back in the mobile app. It didn't do anything.

[zsolt-donca]

And the weirdest thing is that I did the same setup on my wife's phone (the same model as mine, same app version, different immich user), and her photos are not duplicated, and everything seems to have the cloud with the checkmark icon in it, as it should.

[alextran1502]

Where did the assets with the cloud icon get uploaded? the asset with the cross checkmark item meaning they haven't been backup yet?

[zsolt-donca]

I just picked one file named PXL_20230625_134316938.jpg, the remote path is:

/var/local/immich/store/library/d30dd4c8-53c2-4f26-81f2-f7b743ac76e0/2023/2023-06-25

The "duplicate" shows up as having the same name. I just checked the checksums of the two files, and even though they have the same size (4106854 bytes), the md5 and sha1 checksums are different. Is this intended behavior? Or it this difference behind the issue?

[alextran1502]

From your screenshot, there are two icons

• The crossed cloud means it is on the device, local, and not yet backed up.
• The cloud with a check mark means that it is on the device and has been uploaded.

From the step you outlined for reproduction, you haven't performed backing up on this device, is that correct?

[zsolt-donca]

I'm sorry about this confusion. The icons on the screenshot you cropped are like that because I hit the "download" button on two images with the cloud icons. I didn't do the same on the other pictures, and they look like all pairs looked like initially: one with the cloud icon, and another with the crossed out cloud icon.

Before writing these lines I tested the same thing, and the cloud-iconed photos completely disappeared from the gallery. 🫤

[alextran1502]

Can you help me with the question of whether or not you have uploaded anything photos or videos on the phone that you used to take the screenshot?

[zsolt-donca]

The photos on the screenshot were uploaded using the "Backup Albums" feature from the phone that I took the screenshot from. My photo album was initially empty, I selected some albums to back up, and then those duplicates practically immediately showed up (that is, each photo twice, one with cloud icon, another one with the cloud striked through).

[alextran1502]

Thank you.

For photos with the cloud-only icon, it means they exist on the server but not on the device. Can you confirm that you haven't run the upload/backup for any assets on this device?

Can you help with a screenshot of the backup page?

Also, can you help me enable the advanced debugging on the settings pagee? Then for the pair of duplicated assets, capture the screenshot of the info page of each image. It should show the file information to see if there is something different between those two files.

[zsolt-donca]

Can you confirm that you haven't run the upload/backup for any assets on this device?

I am sure that the issue appeared before doing anything more than selecting some albums to back up. Since then I tried fumbling around with the settings, as it wasn't clear to me that this is likely a bug that I should report. I remember having changed some Backup settings and then pressing the "Start Backup" icon.

[zsolt-donca]

Here are some screenshots of the Backup page:

[zsolt-donca]

Also, can you help me enable the advanced debugging on the settings pagee?

How do I do that?

[zsolt-donca]

Here are screenshots of the info pages of a pair of identical images:

And the images look like this in the gallery:

[zsolt-donca]

I think you were referring to these info, the local one:

{
  "id": 24,
  "remoteId": "N/A",
  "localId": "1000002286",
  "checksum": "3i0lGXxH535ZHavulFW9QcWe/YQ=",
  "ownerId": 7001128083095369042, 
  "livePhotoVideoId": "N/A",
  "fileCreatedAt": "2023-06-25 14:47:31.000",
  "fileModifiedAt": "2023-06-25 14:47:34.000", 
  "updatedAt": "2023-06-25 14:47:34.000", 
  "durationInSeconds": 0, 
  "type": "AssetType.image",
  "fileName": "PXL_20230625_114731056.jpg", 
  "isFavorite": false, 
  "isRemote: false,
  "storage": "AssetState.local",
  "width": 3072,
  "height": 4080,
  "isArchived": false
}

And the remote one:

{
  "id": 399,
  "remoteId": "e409707e-3c28-4e8e-b6a1-f34a8a6c5fe5",
  "localId": "N/A",
  "checksum": "mALNwZE1ZqEUGh8/LseVNwYaPF8=",
  "ownerId": 7001128083095369042, 
  "livePhotoVideoId": "N/A",
  "fileCreatedAt": "2023-06-25 14:47:31.000",
  "fileModifiedAt": "2023-06-25 17:47:34.000", 
  "updatedAt": "2023-06-26 12:23:08.768", 
  "durationInSeconds": 0, 
  "type": "AssetType.image",
  "fileName": "PXL_20230625_114731056.jpg", 
  "isFavorite": false, 
  "isRemote: true,
  "storage": "AssetState.remote",
  "width": 3072,
  "height": 4080,
  "isArchived": false
}

[alextran1502]

Hmm, you can see the updatedAt property is different on the remote one. Did the file get modified once it get to UPLOAD_LOCATION?

[zsolt-donca]

Yeah, I saw that too. The checksum is different also. I didn't do anything to any of the pictures. I have some tens of cases, spanning across at least three days. I don't know what could have cause these changes.

[alextran1502]

@fyfrey Can you think of any scenario in this case that could happen?

[fyfrey]

This is why I made the PR to find and delete corrupt assets.

The assets uploaded miss their EXIF Info. Google thought it would be cool to randomly strip GPS EXIF from files :-). Same size but different hash. Since we now sync/merge assets based on hash, you have duplicates..

Solution: Delete all remote assets and reupload with newest app (1.64.0) release. If that is a no-go, wait for my PR to delete only corrupt uploaded assets.

[zsolt-donca]

This conclusion is fine by me. I will delete the remote assets and re-upload everything. 👍

Google thought it would be cool to randomly strip GPS EXIF from files

This is the most Google thing that I heard in a while... 🫣

[zsolt-donca]

Thanks everyone for this great project!

[fyfrey]

Please let me know if it works (or still produces files with different hashes). Thanks!

[zsolt-donca]

It works. I installed the latest release (both on mobile and on the server), cleaned up the duplicate photos on web, backed up the photos once again locally, and this time there were no duplicates. Also gave the mobile app location permissions for good measure.