[BUG] "STORAGE TEMPLATE MIGRATION" job makes duplicates in library

[thierrybla]

The bug

After using the "STORAGE TEMPLATE MIGRATION" job it makes quite a lot of duplicates.

The OS that Immich Server is running on

Docker

Version of Immich Server

v1.56.2

Version of Immich Mobile App

v1.56.2

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

version: "2.1"
services:
  immich:
    image: ghcr.io/imagegenius/immich:latest
    container_name: immich
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Etc/UTC
      - DB_HOSTNAME=192.168.1.x
      - DB_USERNAME=postgres
      - DB_PASSWORD=postgres
      - DB_DATABASE_NAME=immich
      - REDIS_HOSTNAME=192.168.1.x
      - DB_PORT=5432
      - REDIS_PORT=6379
      - REDIS_PASSWORD=
    volumes:
      - path_to_appdata:/config
      - path_to_photos:/photos
      - path_to_machine-learning:/config/machine-learning #optional
    ports:
      - 8080:8080
    restart: unless-stopped

Your .env content

N/A

Reproduction steps

1. Upload photos with random names
2. Click "STORAGE TEMPLATE MIGRATION" job
3. Look in your library and see duplicates

Additional information

No response

[thierrybla]

Photo before the job

and after:

They are both in my library at this moment.

[alextran1502]

Hello, did you change the path when you ran the migration job or did you keep the old path? let me know and I can help taking a look

[thierrybla]

Hi! Thanks for the quick answer! I only changed the file name.

[alextran1502]

@thierrybla Can you elaborate on changing the file name? Can you show the exact value changed in the storage migration box?

[alextran1502]

Please note that any manual file system level operation will not applied to Immich because it causes the mismatch with the Immich's database

[thierrybla]

This is basiclly what I did and then ran the job.

[thierrybla]

Please note that any manual file system level operation will not applied to Immich because it causes the mismatch with the Immich's database

I didn't change anything manually.

[thierrybla]

@alextran1502 do you have time to debug this?

[alextran1502]

I did try and run the migration with the same template you specified but couldn't reproduce the issue you are having 🤔 . Can you try on the new release version?

[thierrybla]

I did try and run the migration with the same template you specified but couldn't reproduce the issue you are having 🤔 . Can you try on the new release version?

Should I start from 0? Do you have any way to remove the created duplicates?

[alextran1502]

Besides manually deleting them on the web, we don't have any other way. I am not sure about this bug, seems strange

[thierrybla]

Besides manually deleting them on the web, we don't have any other way. I am not sure about this bug, seems strange

What checks does immich do for duplicates? Seems having a different name is already enough to upload the same file twice?

Thank you for the swift replies :).

[jrasm91]

A duplicate in Immichs means byte for byte identical files. If that's not the case with your files, then it's working as designed. We hope to improve this in the future.

[thierrybla]

A duplicate in Immichs means byte for byte identical files. If that's not the case with your files, then it's working as designed. We hope to improve this in the future.

Is this only upon upload or is there a background job for this as well?

[jrasm91]

We calculate the file hash on upload and store it in the database. There is a unique key constraint on the column that makes it impossible for two files with the same hash to exist (for a single user). So uploading the same file again would not create a new record.

[thierrybla]

We calculate the file hash on upload and store it in the database. There is a unique key constraint on the column that makes it impossible for two files with the same hash to exist (for a single user). So uploading the same file again would not create a new record.

Hmm I see, still weird how Immich somehow made a duplicate of almost all my library files after running the job..

@alextran1502 I have a zip attached with the original photo and the photo that immich duplicated, maybe put the original photo called "2023-05-18-IMG-20230518-WA0000.jpg" into your own Immich and run the job yourself. It might be interesting to see if it also dupes it for you.

I will start over from scratch for myself.. this time I will right away change the template to something I want and not change it after the uploads are done. 2 photos.zip

[alextran1502]

Upon looking at the assets in the file browser, looks like they are two different version

The date modified and the size are different

[thierrybla]

Upon looking at the assets in the file browser, looks like they are two different version

The date modified and the size are different

I can assure you that I didn't manually edit these files myself or used a different program to do anything to it after upload. Could immich change the things itself?

[thierrybla]

To add, I uploaded the image with the CLI tool on the 20th and used the rename job on the 21th.

[alextran1502]

Immich doesn't touch the original file beside moving them to the template you specify. Can you start fresh and upload a few files and then try to move them again? Can you help record the change in the template?

[thierrybla]

I will upload some images when I start fresh and try to reproduce what I did. I will report back here once I have done that.

[thierrybla]

For now I don't experience it, I will upload all my photos and see if something happens.. For now I will close this issue and reopen when I experience it again.