search

simulot / immich-go

An alternative to the immich-CLI command that doesn't depend on nodejs installation. It tries its best for importing google photos takeout archives.
GNU Affero General Public License v3.0
1.63k stars 49 forks source link

Takout zip is unsupported file type #357

Closed CloneXpert closed 2 months ago

CloneXpert commented 3 months ago

I hope I'm not missing something obvious, but just running the basic command on my standard takeout zip files sees them all as unsupported files:

v0.18.1: immich-go.exe -server=http://192.168.18.111:2283/ -key=<my key> upload -create-albums -google-photos "g:\GoogleTakeout\takeout-*.zip" 

 INFO | Browsing google take out archive... | time="2024-07-06T02:53.17 CEST"
 INFO | unsupported file | file=takeout-20240701T221728Z-004.zip reason="unsupported file type" time="2024-07-06T02:53.17 CEST"
...

immich-go_2024-07-06_14-53-17.log

simulot commented 3 months ago

I have tested on a windows 11 machine, with 0.18.1 and 0.18.2 versions. Both are accepting the following command:

immich-go -server=http://immich2283 -key=<key> upload -google-photos -dry-run "C:\Users\dev\test\takeout-*.zip"
CloneXpert commented 3 months ago

Thanks for checking.

Are you handling localization (hu specifically): the path inside my takeout can look like this, e.g.:

takeout-20240701T221729Z-005.zip\Takeout\Google Fotók\Nem mentett dokumentum(24)\metaadatok.json

I wonder if that's causing the unsupported file type?

simulot commented 3 months ago

Glad you talk about localisation...

So far, I have noticed:

So the .json file is read to determine its type album, photo, print order, whatever... We can thank Google to made the takeout horribly complex...

file=takeout-20240701T221728Z-004.zip reason="unsupported file type"

This shows the that immich go takes zip files as if their were photos. Could open a command windows and type : 

dir "g:\GoogleTakeout\takeout-*.zip"
CloneXpert commented 3 months ago

C:\Users\Tamas>dir "g:\GoogleTakeout\takeout-*.zip" Volume in drive G is ST8000DM004-2 Volume Serial Number is 2697-2907

Directory of g:\GoogleTakeout

07/03/2024 10:15 PM 53,690,142,444 takeout-20240701T221728Z-004.zip 07/02/2024 10:11 PM 53,696,612,309 takeout-20240701T221729Z-001.zip 07/02/2024 10:00 PM 53,690,484,456 takeout-20240701T221729Z-002.zip 07/03/2024 12:33 AM 53,691,850,828 takeout-20240701T221729Z-003.zip 07/03/2024 09:19 PM 53,690,679,934 takeout-20240701T221729Z-005.zip 07/03/2024 10:17 PM 53,689,056,409 takeout-20240701T221729Z-006.zip 07/04/2024 12:38 AM 53,688,844,356 takeout-20240701T221729Z-007.zip 07/04/2024 08:35 PM 53,688,798,874 takeout-20240701T221729Z-008.zip 07/04/2024 09:30 PM 53,683,060,522 takeout-20240701T221729Z-009.zip 07/04/2024 08:26 PM 53,686,026,715 takeout-20240701T221729Z-010.zip 07/04/2024 11:04 PM 7,143,560,679 takeout-20240701T221729Z-011.zip 11 File(s) 544,039,117,526 bytes 0 Dir(s) 5,618,466,869,248 bytes free

simulot commented 3 months ago

So far, the only way I have found to reproduce same message is with this command:

immich-go -server=http://immich2283 -key=<key> upload -google-photos -dry-run "C:\Users\dev\test"

which is unexpected too

CloneXpert commented 3 months ago

Could I provide you with any additional info/data?

simulot commented 2 months ago

I have retested with the latest immich-go release and Windows11 , and this command line works as exprected:

immich-go.exe -server=http://192.168.18.111:2283/ -key=<my key> upload -create-albums -google-photos "g:\GoogleTakeout\takeout-*.zip"
CloneXpert commented 2 months ago

Sorry for my late reply. I retested with the new version 0.19.1 and am getting the same results. I'm running it from Win 10 Pro / powershell.

CloneXpert commented 2 months ago

I debugged the main branch quickly and I think I found the culprit... it seems like zip archive number 2/11 is corrupt, even though it looked like it's a valid 50GB zip. I seen it happen in

for _, f := range files { fsys, err := zip.OpenReader(f) if err != nil { errs = errors.Join(errs, err)

with err: "zip: not a valid zip file" and exits.

Unfortunately, my takeout request has expired, so I can no longer redownload this one. It feels pretty silly having to redownload the whole 563GB once again...

simulot commented 2 months ago

Thank for your investigation. I'll clarify the error message

CloneXpert commented 2 months ago

I downloaded another 511GB of takeout and tried to upload again. Once again, I'm getting the same behavior as above, but this time I checked there are no corrupted part (zip) files.

here's the output:

PS C:\Users\Tamas\Downloads\immich-go> .\immich-go.exe -server=http://192.168.18.111:2283/ -key= upload -create-albums -google-photos "g:\GoogleTakeout\takeout-*" immich-go 0.20.1, commit 9a48344615138c11f637b952be15bde8711a2892, built at 2024-07-16T17:27:12Z . . | || | || | ||(| | ─ (|() v 0.20.1 )

Input analysis:

scanned image file : 0 scanned video file : 0 scanned sidecar file : 0 discarded file : 0 unsupported file : 11 file duplicated in the input : 0 associated metadata file : 0 missing associated metadata file : 0

Uploading:

uploaded : 0 upload error : 0 file not selected : 0 server's asset upgraded with the input : 0 server has same asset : 0 server has a better asset : 0

Check the log file: C:\Users\Tamas\AppData\Local\immich-go\immich-go_2024-07-30_19-04-41.log

simulot commented 2 months ago

I don't understand how this can happen. On the other hand, I haven't yet released the fix #374

CloneXpert commented 2 months ago

I realized the old master checkout I used for debugging now does work. I would just need to have proper executable for have faster/better console output.

Update: Ok, found once again what was wrong. I had the path: "g:\GoogleTakeout\takeout-*" without the zip extension. In VSCode while debugging I had it with the extension: "g:\GoogleTakeout\takeout-*.zip"

I think this is a bug, because it correctly expands the filenames to all the 11 zip files, but yet treats them as an error. Just having the zip extension makes it work.

simulot commented 2 months ago

Got it! Thanks