simulot / immich-go

An alternative to the immich-CLI command that doesn't depend on nodejs installation. It tries its best for importing google photos takeout archives.
GNU Affero General Public License v3.0
1.76k stars 56 forks source link

How can I know if the upload process worked without erros? #18

Closed Deses closed 1 year ago

Deses commented 1 year ago

I ran an upload job and the final lines were those:

←[33m21 server assets to delete.←[39m
←[32m21576 media scanned, 15724 uploaded.←[39m
←[32mDone.←[39m

What does it mean that 21576 files were scanned and only 15724 got uploaded? My Immich instance is reporting that there are 14.909 photos and 796 videos, so 15705 total files. A discrepancy of 19 to 5871 files.


BTW, I ran your app on Windows and it worked very well, but I really just ran it once so that's not very in-depth testing. :P Maybe something that worked for you in Linux but it's not working on Window's cmd is the color formatting, which I guess ←[32m and ←[39m are trying to be.

Deses commented 1 year ago

Nevermind, I took a deeper look at the cmd window and found many like these:

←[32mUploading "Takeout/Google Fotos/Photos from 2021/IMG_20210327_113618_351-01_wm.jpg"...←[31mError: AssetUpload, POST, http://immich.unraid.nas/api/asset/upload
Post "http://immich.unraid.nas/api/asset/upload": dial tcp: lookup immich.unraid.nas: no such host

It's weird that it failed with no such host when it wasn't complaining about it for most of the files. Maybe it's safer to use <IP:PORT> after all.

Update: <IP:PORT> is, indeed, more reliable. It pulled 3404 files more.


As a feature request, I'd like to have one switch that toggles logging, so all the console output goes to a file. I know I could use 1> immich-go.log 2>&1 but doesn't seem very elegant. :)

simulot commented 1 year ago

What does it mean that 21576 files were scanned and only 15724 got uploaded? My Immich instance is reporting that there are 14.909 photos and 796 videos, so 15705 total files. A discrepancy of 19 to 5871 files.

There are 2 explanations for the differences:

1/ The takeout archive contains lot of duplicates. Photos that belong to an album are stored in the album folder, and in the year folder. Immich-go scans both, and upland just one.

I use the file name and the date of capture to identify the duplicates. It's immediate and seams to be good enough.

2/ immich server use hash to detect duplicates. On the server side Immich take the SHA1 (i think) of the incoming file an compare it to its database. When you have renamed a file, my test on the name doesn't work. The file is uploaded but rejected by the server.

Maybe something that worked for you in Linux but it's not working on Window's cmd is the color formatting, which I guess ←[32m and ←[39m are trying to be.

This depends on witch version of the terminal you are using. I heard it's now possible to have a modern terminal in windows that understand ANSI colors.

Anyway, you can disable the colors by adding -no-colors-log flag beside the -server' option, before theupload` command.

←[32mUploading "Takeout/Google Fotos/Photos from 2021/IMG_20210327_113618_351-01_wm.jpg"...←[31mError: AssetUpload, POST, http://immich.unraid.nas/api/asset/upload Post "http://immich.unraid.nas/api/asset/upload": dial tcp: lookup immich.unraid.nas: no such host

Hum... At program start, the program tests the server's api doing a get http://immich.unraid.nas/api/server-info/ping and stops if the server doesn't respond. You have this error during upload, this means that the ping was success full, hence the server name is correctly resolved at least one time.

For sure the work around is to you the IP:PORT in the url.

The name resolution is faulty here. But I can't reproduce this on my system, linux, no local name server.

I have 2 hypothesis in mind:

  1. There is a antivirus / pi-hole / whatever that refuse the name query
  2. The program doesn't behave correctly under windows.

I don't know yet how investigate this case. Let me know if you are able to make a test with a specially crafted binary exe?

As a feature request, I'd like to have one switch that toggles logging, so all the console output goes to a file. I know I could use 1> immich-go.log 2>&1 but doesn't seem very elegant. :)

I'm using immich-go ... | tee immich-go.log, of course the linux way.

Any way, i'll add an option to write logs into a given file.

Deses commented 1 year ago

Thank you for the detailed explanation!

When the other issues are resolved I'll clear the photo library and upload the takeout again and compare if everything got uploaded correctly on the first try.

This depends on witch version of the terminal you are using. I heard it's now possible to have a modern terminal in windows that understand ANSI colors.

I used good ol' cmd, and the terminal you are talking about is Windows Terminal, which I should have been using by now anyway... I just use PuTTy more regularly and never really bothered with anything better for Windows itself lol

There is a antivirus / pi-hole / whatever that refuse the name query

Indeed I have a PiHole, but I did add a DNS record pointing the hostname I'm using locally to the IP, so I don't know... Anything could be at fault here. For sure the best solution is to just use the IP.

simulot commented 1 year ago

Indeed I have a PiHole, but I did add a DNS record pointing the hostname I'm using locally to the IP, so I don't know... Anything could be at fault here. For sure the best solution is to just use the IP.

Some of those days, I'll do a test with in a virtual windows machine with a name resolution done by pi-hole...

Deses commented 1 year ago

If you want to save the hassle I can always run any tests you want me to run.