The API accept duplicated files

[simulot]

The bug

The immich-go project has received few strange reports telling that when uploading another time the same collection, few files are still uploaded. ex: https://github.com/simulot/immich-go/issues/359

After analyse it appears that the API endpoint replies success when uploading a file with a different name, but with the same content (same SHA1).

The OS that Immich Server is running on

Mint, windows

Version of Immich Server

v1.107.2

Version of Immich Mobile App

N/A

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

Not relevant

Your .env content

Not relevant

Reproduction steps

Reproduced on a brand new docker instance with 2 files having the same SHA1 (g859jTnWLXA2WuwLhbv96fjHBV4= or \x83ce7d8d39d62d70365aec0b85bbfde9f8c7055e)

1. Upload File 1 -> OK
2. Upload File 2 -> OK

Relevant log output

---- CONTAINER LOGS ----
immich_postgres          | 2024-07-07 17:51:58.968 UTC [178] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
immich_postgres          | 2024-07-07 17:51:58.968 UTC [178] DETAIL:  Key ("ownerId", checksum)=(d0af84ae-8067-4551-8c4b-5f3c41fb4190, \x83ce7d8d39d62d70365aec0b85bbfde9f8c7055e) already exists.
immich_postgres          | 2024-07-07 17:51:58.968 UTC [178] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath", "createdAt", "updatedAt", "deletedAt", "fileCreatedAt", "localDateTime", "fileModifiedAt", "isFavorite", "isArchived", "isExternal", "isOffline", "checksum", "duration", "isVisible", "livePhotoVideoId", "originalFileName", "sidecarPath", "stackId", "duplicateId") VALUES (DEFAULT, $1, $2, $3, $4, $5, $6, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $7, $8, $9, $10, $11, DEFAULT, $12, $13, $14, $15, DEFAULT, $16, $17, DEFAULT, DEFAULT) RETURNING "id", "thumbnailPath", "encodedVideoPath", "createdAt", "updatedAt", "deletedAt", "isFavorite", "isArchived", "isExternal", "isOffline", "isVisible"

--- IMMICH-GO API TRACE --- 
2024-07-07T19:51:58+02:00 QUERY 5 AssetUpload POST http://localhost:2283/api/assets
   Accept [application/json]
   Content-Type [multipart/form-data; boundary=888c00fb943196e0e009f387776b7f2f4e6d8a8c27561ebd2617add58d12]
   X-Api-Key [redacted]
-- Empty body or binary body not dumped --
2024-07-07T19:51:58+02:00 RESPONSE 5 AssetUpload POST http://localhost:2283/api/assets
-- response body --
{
 "id": "00c0c6ae-2223-4b44-a154-e06c14e034a1",
 "duplicate": false
}
-- response body end --

2024-07-07T19:51:58+02:00 QUERY 6 AssetUpload POST http://localhost:2283/api/assets
   Accept [application/json]
   Content-Type [multipart/form-data; boundary=2d0f165aac48af17ae42028715a9ac574c5c548e7a1d94bfeaf6cdddb406]
   X-Api-Key [redacted]
-- Empty body or binary body not dumped --
2024-07-07T19:51:58+02:00 RESPONSE 6 AssetUpload POST http://localhost:2283/api/assets
-- response body --
{
 "id": "00c0c6ae-2223-4b44-a154-e06c14e034a1",
 "duplicate": false
}

Additional information

When dropping the files on the web UI, the UI reports the duplicate error

[jrasm91]

Can you provide steps to reproduce this? A few things that are confusing about this report. The response in the attached logs uses duplicate: true/false, but we're returning status: duplicate/etc instead since a few versions ago. Also we have specific tests to reupload the same file in our e2e. Also, if the web correctly handles it, that implies the API is functioning correctly.

[simulot]

The API get 2 success as shown. And yes the web interface behaves differently

At the end, there is only one sample of the photo in immich.

[bo0tzz]

I believe the success response when uploading a duplicate asset is intended - your goal is for that asset to exist in Immich, and that goal succeeded. I do believe there's something in the response that indicates it was a duplicate though.

[bo0tzz]

The web client I believe hashes the file before uploading, so its reporting does not come from the upload endpoint.

[simulot]

It's a regression. The API used to reply duplicate=true... Note that the returned ID is different... as if the asset has been accepted

[jrasm91]

It isn't a regression. The response format changed.

[simulot]

I haven't noticed that. Thanks