search

simulot / immich-go

An alternative to the immich-CLI command that doesn't depend on nodejs installation. It tries its best for importing google photos takeout archives.
GNU Affero General Public License v3.0
1.63k stars 49 forks source link

Unexpected Discrepancy in 'Server has same quality' Metric After Re-uploading Images #359

Closed ckocyigit closed 3 months ago

ckocyigit commented 3 months ago

I'm just uploading a bunch of media and I ran a command like this:

immich-go -server=https://x.x.de -key=x upload **

After almost 6,000 images were "uploaded," I ran the same command again and expected the "Server has same quality" metric to basically match the count of images in that folder. Surprisingly, this was not the case; only around 1,000 were marked as "same quality." Why is this happening? This makes me feel really insecure about whether the files are actually uploaded.

First Run: grafik

Second Run:

grafik

Third Run:

grafik

simulot commented 3 months ago

This is unexpected. Is the server database updated by another means during that time? (I think to the mobile app)?

It is possible to start over with a fresh immich installation and generate counter files (-debug-counters) and keep the logs?

simulot commented 3 months ago

Are you importing a google takeout archive? I can't reproduce your result with a small takeout file (480 files)

simulot commented 3 months ago

I did a test on a bigger data set: The only strange figure is the duplicated in the input.

immich-go  dev, commit dafec49d62b589cf02c989a5cb53bb84ff739125, built at 2024-07-07T09:01:38Z
. _ _  _ _ . _|_     _  _ 
|| | || | ||(_| | ─ (_|(_)
               v dev _)   

Input analysis:
---------------
scanned image file                      :    4152
scanned video file                      :     256
scanned sidecar file                    :    4405
discarded file                          :       0
unsupported file                        :       1
file duplicated in the input            :     168
associated metadata file                :    4408
missing associated metadata file        :       0

Uploading:
----------
uploaded                                :    4240
upload error                            :       0
file not selected                       :       0
server's asset upgraded with the input  :       0
server has same asset                   :       0
server has a better asset               :       0
immich-go  dev, commit dafec49d62b589cf02c989a5cb53bb84ff739125, built at 2024-07-07T09:01:38Z
. _ _  _ _ . _|_     _  _ 
|| | || | ||(_| | ─ (_|(_)
               v dev _)   

Input analysis:
---------------
scanned image file                      :    4152
scanned video file                      :     256
scanned sidecar file                    :    4405
discarded file                          :       0
unsupported file                        :       1
file duplicated in the input            :     167
associated metadata file                :    4408
missing associated metadata file        :       0

Uploading:
----------
uploaded                                :       0
upload error                            :       0
file not selected                       :       0
server's asset upgraded with the input  :       0
server has same asset                   :    4240
server has a better asset               :       0
ckocyigit commented 3 months ago

Unfortunately I cannot help with the first comment.

Immich runs with the example docker compose structure from the immich repo with the latest additionals to remove the microservice container and change the command from server.

I would be baffled if sometime else in interacting with the databse while I'm uploading. I'm basically the only user on this immich instance.

I do have the mobile app but I cannot combine this fact an the issue I'm having.

no I'm not uploading a google takeout archive it's just plain files

I just did a bigger test with a completely different dataset:

Second run:

immich-go  0.19.0, commit dafec49d62b589cf02c989a5cb53bb84ff739125, built at 2024-07-07T09:01:38Z
. _ _  _ _ . _|_     _  _
|| | || | ||(_| | ─ (_|(_)
            v 0.19.0 _)

Input analysis:
---------------
scanned image file                      :    6561
scanned video file                      :    2215
scanned sidecar file                    :       0
discarded file                          :       0
unsupported file                        :       2
file duplicated in the input            :       0
associated metadata file                :       0
missing associated metadata file        :       0

Uploading:
----------
uploaded                                :    7751
upload error                            :       0
file not selected                       :       0
server's asset upgraded with the input  :       0
server has same asset                   :    1017
server has a better asset               :       8

Third run:

immich-go  0.19.0, commit dafec49d62b589cf02c989a5cb53bb84ff739125, built at 2024-07-07T09:01:38Z
. _ _  _ _ . _|_     _  _
|| | || | ||(_| | ─ (_|(_)
            v 0.19.0 _)

Input analysis:
---------------
scanned image file                      :    6561
scanned video file                      :    2215
scanned sidecar file                    :       0
discarded file                          :       0
unsupported file                        :       2
file duplicated in the input            :       0
associated metadata file                :       0
missing associated metadata file        :       0

Uploading:
----------
uploaded                                :    7731
upload error                            :       0
file not selected                       :       0
server's asset upgraded with the input  :       0
server has same asset                   :    1037
server has a better asset               :       8

Check the log file:  /root/.cache/immich-go/immich-go_2024-07-07_15-15-48.log

The immich instance is running behind a traefik reverse proxy but I can not imagine that this may have any implications.

simulot commented 3 months ago

Did you check the result in the Immich app? I suspect a problem in counters.

ckocyigit commented 3 months ago

How would I check?

simulot commented 3 months ago

image

I'm available on Discord @simulot

simulot commented 3 months ago

There is a problem in immich when uploading duplicated files with a different name. The Upload API accept all copies of the same file, giving a success reply for all of them. But only one of them is actually uploaded.

At the second attempt, immich-go detect the 1st occurrence as already uploaded, and count it as "exists on the server". The next occurrence of the file isn't detected as duplicate by Immich-go is uploaded, and once again the API accept it.

Here is the immich-go API trace:

2024-07-07T19:25:47+02:00 QUERY 5 AssetUpload POST http://localhost:2283/api/assets
   Accept [application/json]
   Content-Type [multipart/form-data; boundary=845b8785d3c1de86069b7d8795a177c46b5b1331055d957e55b2de4dc208]
   X-Api-Key [redacted]
-- Empty body or binary body not dumped --
2024-07-07T19:25:47+02:00 RESPONSE 5 AssetUpload POST http://localhost:2283/api/assets
-- response body --
{
 "id": "176a112e-d1f6-40f5-ad6d-7ceaa202c91e",
 "duplicate": false
}
2024-07-07T19:25:55+02:00 QUERY 7 AssetUpload POST http://localhost:2283/api/assets
   Accept [application/json]
   Content-Type [multipart/form-data; boundary=fd29004d9847c38e94abae677c3f3296aee4fb572b4d2a92146d58a073d6]
   X-Api-Key [redacted]
-- Empty body or binary body not dumped --
2024-07-07T19:25:55+02:00 RESPONSE 7 AssetUpload POST http://localhost:2283/api/assets
-- response body --
{
 "id": "176a112e-d1f6-40f5-ad6d-7ceaa202c91e",
 "duplicate": false
}

The immich logs shows:

immich_postgres          | 2024-07-07 17:37:01.448 UTC [143] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
immich_postgres          | 2024-07-07 17:37:01.448 UTC [143] DETAIL:  Key ("ownerId", checksum)=(d0af84ae-8067-4551-8c4b-5f3c41fb4190, \x83ce7d8d39d62d70365aec0b85bbfde9f8c7055e) already exists.
immich_postgres          | 2024-07-07 17:37:01.448 UTC [143] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath", "createdAt", "updatedAt", "deletedAt", "fileCreatedAt", "localDateTime", "fileModifiedAt", "isFavorite", "isArchived", "isExternal", "isOffline", "checksum", "duration", "isVisible", "livePhotoVideoId", "originalFileName", "sidecarPath", "stackId", "duplicateId") VALUES (DEFAULT, $1, $2, $3, $4, $5, $6, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $7, $8, $9, $10, $11, DEFAULT, $12, $13, $14, $15, DEFAULT, $16, $17, DEFAULT, DEFAULT) RETURNING "id", "thumbnailPath", "encodedVideoPath", "createdAt", "updatedAt", "deletedAt", "isFavorite", "isArchived", "isExternal", "isOffline", "isVisible

the 83ce7d8d39d62d70365aec0b85bbfde9f8c7055e is the SHA1 hash of the 2 files.