search

simulot / immich-go

An alternative to the immich-CLI command that doesn't depend on nodejs installation. It tries its best for importing google photos takeout archives.
GNU Affero General Public License v3.0
1.93k stars 55 forks source link

feat: add retry after temporary network failure #113

Open anoadragon453 opened 11 months ago

anoadragon453 commented 11 months ago

While attempting to upload photos from my Google takeout, I see the following error for about 1 in every 20 photos:

Server error             : Takeout/Google Photos/Photos from 2014/IMG_5170.JPG: AssetUpload, POST, https://i.amorgan.xyz/api/asset/upload
Post "https://i.amorgan.xyz/api/asset/upload": remote error: tls: bad record MAC

This is annoying as it appears that these photos are just being left behind during the upload. I'll have to note down which photos failed and try them again, repeating the process until there are no errors left.

On my immich microservices container, I am seeing the following error regularly appear:

Dec 30 22:05:37 plonkie docker-immich_microservices-start[1341286]: [Nest] 6  - 12/30/2023, 9:05:37 PM    WARN [Error: File not found - /usr/src/app/upload/upload/898fa47d-ff65-4b82-978b-ac59e0560b42/f1926aac-fa61-4908-a3bf-4aaa27c62388.JPG.xmp
Dec 30 22:05:37 plonkie docker-immich_microservices-start[1341286]:     at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:43:29)
Dec 30 22:05:37 plonkie docker-immich_microservices-start[1341286]:     at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)] Error reading exif data (upload/upload/898fa47d-ff65-4b82-978b-ac59e0560b42/f1926aac-fa61-4908-a3bf-4aaa27c62388.JPG.xmp): Error: File not found - /usr/src/app/upload/upload/898fa47d-ff65-4b82-978b-ac59e0560b42/f1926aac-fa61-4908-a3bf-4aaa27c62388.JPG.xmp

as well as:

Dec 30 22:05:37 plonkie docker-immich_microservices-start[1341286]: [Nest] 6  - 12/30/2023, 9:05:37 PM    WARN [MetadataService] Exif data has latitude and longitude of 0, setting to null

which may be related? Edit: These errors are appearing for every uploaded file, not just those that fail to upload. Thus it appears that they're unrelated.

The full immich-go command I am using is:

.\immich-go.exe -server https://i.amorgan.xyz -key [redacted] -no-colors-log -log-file=takeout-upload-log upload -google-photos .\takeout-*.zip
immich-go  0.9.6, commit 1782249564abb77f01780282ca4d8ddaf7ef6f69, built at 2023-12-30T08:30:25Z

The same occurs with the -skip-verify-ssl flag. This is on Windows 11 x86_64 using the v0.9.6 release exe. This was the final output:

Create the album Untitled
Scan of the sources:
 12922 files in the input
--------------------------------------------------------
  6405 photos
    70 videos
  6443 metadata files
  6475 files with metadata
     4 discarded files
     0 files having a type not supported
     0 discarded files because in folder failed videos
 12922 input total (difference 0)
--------------------------------------------------------
  5802 uploaded files on the server
     0 upgraded files on the server
   372 files already on the server
     0 discarded files because of options
     2 discarded files because duplicated in the input
     0 discarded files because server has a better image
   299 errors when uploading
  6475 handled total (difference 0)
Done.
simulot commented 11 months ago

Thanks for your report.

remote error: tls: bad record MAC

I can't really tell what this means.Immich go uses the very standard library provided by the go language, with default settings.

A google search https://www.google.com/search?q=golang++remote+error%3A+tls%3A+bad+record+MAC give some information:

https://github.com/googleapis/google-cloud-go/issues/1581#issuecomment-531817165

bad record MAC is a very internal error, which usually means network corruption of some sorts (like a proxy changing the contents of the TCP stream), or a severe issue in the TLS stack.

The most useful thing for debugging the latter would be an issue on golang.org/issues with a PCAP and the result of setting tls.Config.KeyLogWriter during that network capture.

Other kind of responses:

I would suggest to change some conditions to learn more on the source of the problem

xmp error

Error reading exif data (...): Error: File not found - .....JPG.xmp

the file isn't found... maybe because of the network error

anoadragon453 commented 11 months ago

Thank you for your in-depth research! I'll give those alternative suggestions a go.

Otherwise, do you think it may be worth adding retries for failed uploads, in case of temporary network failure or other unexpected condition? Even something like max 3 retries per file would probably cover whatever issue I'm having. And it would help add general resilience to the upload process.

simulot commented 11 months ago

Otherwise, do you think it may be worth adding retries for failed uploads, in case of temporary network failure or other unexpected condition? Even something like max 3 retries per file would probably cover whatever issue I'm having. And it would help add general resilience to the upload process.

Sure, added to the todo list