Dry run and real/wet run show different results

GuoxiongXie commented 8 months ago

Hi simulot and team, first of all, thanks a lot for your useful tool! I tried to upload a directory that contains both photos and videos to immich. I first uploaded them through the new immich cli, but some files fail to be uploaded. Immich server stats on admin page shows that I have 24441 photos and 13852 videos on immich server. Then I came across your useful tool and decided to pick up the missing files using your tool.

Here are the steps that I reproduced the issue:

Step 1: I first did a dry run using the following command:

./immich-go -server=http://myServer:2283 -key=<myKey> -log-file=/path/dry_run_1.txt -time-zone=America/Los_Angeles upload -dry-run /path/to/my/directory

I checked the logs of dry run, and saw the following:

Server status: OK
Ask for server's assets...
38293 asset(s) received
Browsing folder(s)...
Done.
Uploaded                 : encoded-video/a0029faf-b017-41cd-94bb-3123b176e147/519572716079.mp4: 519572716079.mp4
Uploaded                 : encoded-video/a0029faf-b017-41cd-94bb-3123b176e147/324353734223.mp4: 324353734223.mp4
**.... <3184 uploaded loglines omitted here> ....**

Managing albums
Scan of the sources:
 41502 files in the input
--------------------------------------------------------
 24484 photos
 16997 videos
     0 metadata files
     0 files with metadata
     0 discarded files
    21 files having a type not supported
     0 discarded files because in folder failed videos
 41502 input total (difference 0)
--------------------------------------------------------
  **3184 uploaded files on the server**
     0 upgraded files on the server
 **38297 files already on the server**
     0 discarded files because of options
     0 discarded files because duplicated in the input
     0 discarded files because server has a better image
     0 errors when uploading
 41481 handled total (difference 0)
Done.

It says "3184 uploaded files on the server", I assume that means there are 3184 missing files that will be uploaded to immich server when I issue a real/wet run. It also reports that "38297 files already on the server".

Step 2: wet/real run

I ran the same command as Step 1 without -dry-run flag:

./immich-go -server=http://myServer:2283 -key=<myKey> -log-file=/path/dry_run_1.txt -time-zone=America/Los_Angeles upload /path/to/my/directory

I saw the following logs (full log, I didn't omit anything below; there's no "Uploaded" log lines emitted):

Server status: OK
Ask for server's assets...
38293 asset(s) received
Browsing folder(s)...
Done.
Managing albums
Scan of the sources:
 41502 files in the input
--------------------------------------------------------
 24484 photos
 16997 videos
     0 metadata files
     0 files with metadata
     0 discarded files
    21 files having a type not supported
     0 discarded files because in folder failed videos
 41502 input total (difference 0)
--------------------------------------------------------
     **0 uploaded files on the server**
     0 upgraded files on the server
 **41481 files already on the server**
     0 discarded files because of options
     0 discarded files because duplicated in the input
     0 discarded files because server has a better image
     0 errors when uploading
 41481 handled total (difference 0)
Done.

I was surprised to see that "0 uploaded files on the server" and "41481 files already on the server". On the other hand, in dry-run (Step 1 above), the tool reported "3184 uploaded files on the server" and "38297 files already on the server". 3184+38297 = 41481. So, it seems like -dry-run actually uploaded those files for me and there's nothing for the wet run to upload. I cross referenced Immich admin page -> Sever Stats. It shows that I have 24441 photos and 13852 videos on immich server. These numbers are the same as the ones that I saw before I ran any immich-go command. So, nothing seems to be uploaded by immich-go tool. There's no error or warning logs, and there's no loglines showing which files were uploaded.

Step 3: dry run again to see if there's still some files that need to upload:

This step is exactly the same as Step 1 above, the same dry run command and the same log output. It reports that "3184 uploaded files on the server" and "38297 files already on the server", which confirms that the wet run in Step 2 didn't actually upload any file to the server.

Any pointer to workaround the issue will be highly appreciated! Thanks in advance for your help!

simulot commented 8 months ago

41481 files already on the server means that immich-go tried to upload those files, but immich has detected them as duplicates.

My guess is that files from your archive has copies of the ones already uploaded, but with different name. To spot them, add -log-level=INFO beside the -log-file and check if you find them on the server.

Here is how duplicate detection works when using immich-go and immich:

immich-go get the list of photos already on the server
immich-go tries to detect logical duplicates using file name and date of capture.
when files with same name, same date of capture are found, immich-go uploads the heaviest one, and discard lighter or replacing the one one on the server by uploading the heaviest one
the immich server calculates the SHA-1 hash of uploaded file, and compare it with its database of SHA-1, whatever the actual file name. If same SHA-1 is found, this is a duplicate files, with same content, and possibly a different name

The dry-run doesn't upload anything, so in this mode the SHA-1 check isn't not done. hence the difference in the report.

GuoxiongXie commented 8 months ago

Wow, your guess is indeed correct! Thanks simulot for your help and clarification!

simulot / immich-go