Closed doegox closed 9 months ago
Failing pics:
Same pics but sanitized EXIF, those work fine.
Side note: because there is no way to get immich-cli printing the pics it's uploading (they only appear briefly in the progress bar), locating such failing pics among a larger set is a bit of a story of finding the needle in the haystack...
Maybe it is not an EXIF problem, I see https://github.com/immich-app/immich/issues/3615 got created since then, mentioning duplicate keys in postgres, and if I check my postgres logs I also see
2023-08-13 12:39:36.737 UTC [110278] ERROR: duplicate key value violates unique constraint "UQ_userid_checksum"
2023-08-13 12:39:36.737 UTC [110278] DETAIL: Key ("ownerId", checksum)=(35755538-1616-4aef-841a-8a0b873fd08b, \xe934a486649279181889266e08eb8aa04dfde96e) already exists.
2023-08-13 12:39:36.737 UTC [110278] STATEMENT: INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "deviceId", "type", "originalPath", "resizePath", "webpPath", "thumbhash", "encodedVideoPath", "createdAt", "updatedAt", "fileCreatedAt", "fileModifiedAt", "isFavorite", "isArchived", "isReadOnly", "checksum", "duration", "isVisible", "livePhotoVideoId", "originalFileName", "sidecarPath") VALUES (DEFAULT, $1, $2, $3, $4, $5, $6, $7, $8, $9, DEFAULT, DEFAULT, $10, $11, $12, $13, $14, $15, $16, $17, DEFAULT, $18, $19) RETURNING "id", "webpPath", "encodedVideoPath", "createdAt", "updatedAt", "isFavorite", "isArchived", "isReadOnly", "isVisible"
and maybe the "EXIF fix" had just the side effect of modifying the file checksum...
If we check the http://immich/api/asset/CLI endpoint there are missing files.
I definitely see that it does NOT return the .MOV
associated with .HEIC
/ .JPG
for iOS 'live' photos. This causes my system to try to upload files over and over.
Second issue, if you check the endpoint, it's actually very risky...
It contains filename + filesize, so if you have 2 files, different name, same directory you'll get a crash as the checksum is filename-independent, yet the id
is based on filename-filesize:
https://github.com/immich-app/CLI/blob/ee750b8d48fce68bd7ada07c540e9d8957e7124e/bin/index.ts#L166
Equally, if you have 2 files with the same name and same size (which honestly isn't that weird of a collision, I have 5 distinct IMG_1999.JPG on my machine) only one will be uploaded.
Update: there is a column that defines whether an item is visible or not (I'm assuming to hide live photos). The asset/:deviceId
does not return invisible items causing the CLI to try to upload 'invisible' items over and over.
Proof of collision, 2 jpgs with different checksum but identical sizes are considered identical.
kristof in ~/image-collision
❮ fd --type file
first/image.jpg
second/image.jpg
kristof in ~/image-collision
❯ fd --type file | xargs sha1sum
ce4de1fa20ab3c6d34034fcd7db29e909d26fc52 first/image.jpg
700b761b3f7584cfc0983a31230aa7d657ff918b second/image.jpg
kristof in ~/image-collision
❮ fd --type file --exec node --eval "const name = process.argv[2]; console.log(path.basename(name) + \"-\" + fs.statSync(name).size)" -
image.jpg-748
image.jpg-748
Second file never gets uploaded because it's already in asset/:deviceId
+1 Also having this issue. Specifically, it seems to happen with HEIC files (?). Unsure.
Hello is this issue still relevance?
Let me test!
Repeated same steps, 2 images, both identical filename-fizesize.jpg. Both get uploaded successfully.
CLI seems to have stopped working though, can't test through there.
We just released a rewrite of the CLI. I'm not sure if it has the same problem or not.
I don't have issues anymore with the new client, thank you !
The bug
Hi, sorry the title may be not clear, not easy to describe in one sentence.
The OS that Immich Server is running on
Docker on Debian
Version of Immich Server
v1.71.0
Version of Immich Mobile App
v1.71.0
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
Use immich-cli upload on a bunch of pics. It uploads them without apparent error. Re-run the same command on the same pics. A few of them want to be re-uploaded, they apparently didn't make their path to the server. Re-run the same command on the same pics again. Same subset again want to be re-uploaded. I'm not sure what the reason is, but when I sanitize the EXIF data with
Then I can upload those pics and if I repeat the command, they are well detected as already present on the server. So I suspect the pics fail on the server because of some EXIF parsing but the error is not reported to the immich-cli.
I selected 2 such pics, before and after EXIF fix, so you can test on your side. I'll try to upload them on the issue else I'll find another way.
Additional information