search

simulot / immich-go

An alternative to the immich-CLI command that doesn't depend on nodejs installation. It tries its best for importing google photos takeout archives.
GNU Affero General Public License v3.0
1.19k stars 35 forks source link

Missing upload files #273

Closed jfreak53 closed 1 month ago

jfreak53 commented 1 month ago

I have 2,385 images and videos in my folder, all MP4's and JPG's. But it only uploaded 2346:

Selection_002

simulot commented 1 month ago

Sounds bad. Have looked at the log file? It may have some clues?

jfreak53 commented 1 month ago

Not sure what to look for in the log file, is there something I can search for in it?

simulot commented 1 month ago

For example, you have noticed that a photo is missing after the import in immich. Search its name in the log, you may have a comment.

jfreak53 commented 1 month ago

I don't know which images are missing, thats just it. The only way I know its missing is the count screenshot I posted above. Counts don't match.

simulot commented 1 month ago

You can search for "error" in the log. Some errors may not been displayed on the status report.

You can also send me the log file on my discord account.

jfreak53 commented 1 month ago

Link to discord user?

jfreak53 commented 1 month ago

Didn't mean to close

simulot commented 1 month ago

simulot

jfreak53 commented 1 month ago

Screenshot_20240531_110736

Only one error in the file from a grep.

simulot commented 1 month ago

Well, it doesn't help so much. I'll check the code. I'll keep you updated

simulot commented 1 month ago

Could you share the log file with me?

simulot commented 1 month ago

I have done some progress on this case. Big files are causing timeouts errors that wasn't correctly reported. This could explain the difference.

jfreak53 commented 1 month ago

Sorry, I'm in a bad signal area for weekend, I can send log tonight if you still need it?

Once I pull the new version, is there a way to only upload missing files and not duplicate things?

simulot commented 1 month ago

Immich-go won't create duplicates. You can start over the download.

Keep me posted with the results. I believe I found the root cause of your problem. Anyway, you can turn on the -log-json option for your test. The produced log is easier to explore.

jfreak53 commented 1 month ago

Still missing 6, but this time it came up with errors :)

Selection_002

INFO | upload error | file=20160813_124834.mp4 error="AssetUpload, POST, http://domain:2283/api/asset/upload\nPost \"http://domain:2283/api/asset/upload\": write tcp 192.168.9.54:51930->0.0.0.0:2283: write: connection reset by peer\n" time="2024-06-02T06:18.39 UTC" INFO | upload error | file=20160903_172910.mp4 error="AssetUpload, POST, http://domain:2283/api/asset/upload\nPost \"http://domain:2283/api/asset/upload\": write tcp 192.168.9.54:51932->0.0.0.0:2283: write: connection reset by peer\n" time="2024-06-02T06:19.00 UTC" INFO | upload error | file=20170210_155743.mp4 error="AssetUpload, POST, http://domain:2283/api/asset/upload\nPost \"http://domain:2283/api/asset/upload\": write tcp 192.168.9.54:51934->0.0.0.0:2283: write: connection reset by peer\n" time="2024-06-02T06:23.26 UTC" INFO | upload error | file=20180105_210806.mp4 error="AssetUpload, POST, http://domain:2283/api/asset/upload\nPost \"http://domain:2283/api/asset/upload\": write tcp 192.168.9.54:51936->0.0.0.0:2283: write: connection reset by peer\n" time="2024-06-02T06:29.33 UTC" INFO | upload error | file=20180410_194715.mp4 error="AssetUpload, POST, http://domain:2283/api/asset/upload\nPost \"http://domain:2283/api/asset/upload\": write tcp 192.168.9.54:51942->0.0.0.0:2283: write: connection reset by peer\n" time="2024-06-02T06:29.50 UTC" INFO | upload error | file=VID_20210925_103459.mp4 error="AssetUpload, POST, http://domain:2283/api/asset/upload\nPost \"http://domain:2283/api/asset/upload\": write tcp 192.168.9.54:51944->0.0.0.0:2283: use of closed network connection\n" time="2024-06-02T06:45.41 UTC"

It seems only videos, so I'd bet still large files. I checked the first file and its 443MB and the last one is 145MB in size.

simulot commented 1 month ago

At least errors are now visible. Apparently the longer timeout has helped too

simulot commented 1 month ago

You can try to drop the missing files on the application web page.

jfreak53 commented 1 month ago

So really strange here, I did try it over web just now. Uploaded just fine. What's strange is, 4 of the 6 were duplicates! So it uploaded 4 of them just fine but never got a response back that it worked. The two that didn't upload were the largest videos, 475MB and 1.4GB's in size respectively.

So definitely a timeout issue. On my side no biggie, I'm done importing and good to go. But just for anyone looking in the future. Thanks!

simulot commented 1 month ago

I have tested on my laptop with 2 big files, 2GB and 8GB... no problem. Works on my machine ;-)