Google takeout upload failed due to missing json files

[swex]

Hello there! trying to upload google takeout archive and bunch of error messages like this:

Error                    : Takeout/Contacts/All Contacts/9219453021086142833.jpg: JSON File not found for this file
Error                    : Takeout/Contacts/All Contacts/Andrey Shef.jpg: JSON File not found for this file

at the end tool said:

Managing albums
Scan of the sources:
128543 files in the input
--------------------------------------------------------
 62430 photos
  5962 videos
 51419 metadata files
 49047 files with metadata
  8491 discarded files
   237 files having a type not supported
     4 discarded files because in folder failed videos
128543 input total (difference 0)
--------------------------------------------------------
     0 uploaded files on the server
     0 upgraded files on the server
     0 files already on the server
 41334 discarded files because of options
  7713 discarded files because duplicated in the input
     0 discarded files because server has a better image
     0 errors when uploading
 49047 handled total (difference 19345)
Done.

and no upload happened. I have 2 suggestions:

1. Maybe only search zip archives in Google photos path? eq: Takeout/Drive/Google Photos because it looks like currently tool checking all takeout paths for images. (I have full takeout backup with all services, not only Google Photos)
2. Add an option to skip those errored images paths.

[simulot]

Immich-go has never been designed to handle contact takeout.

[swex]

I understand that. But it's not contacts takeout, it's all services takeout. Tool probably must search only for Google Photos specific paths, no?

[simulot]

It wasn't a need so far. I think the import process has continued to the end, am I right?

[swex]

no it stops right after error messages, you can see it in result message

 0 uploaded files on the server

[simulot]

At least immich-go should process what it can, and report errors.

[swex]

it does processing and reporting errors but not uploading anything. is it planned behavior?

[simulot]

Not really. I need to check the code

[narucx]

I have the same error but with a lot of videos and photos. Idk if im just doing something wrong here or if those files just arent working

Error                    : Takeout/Google Photos/Photos from 2020/PANO_20200521_163603.vr.jpg: JSON File not found for this file
Error                    : Takeout/Google Photos/Photos from 2020/PANO_20200605_190901.vr.jpg: JSON File not found for this file
Error                    : Takeout/Google Photos/Photos from 2020/VID_20200521_214326.mp4: JSON File not found for this file
Error                    : Takeout/Google Photos/Photos from 2020/VID_20200624_164120.mp4: JSON File not found for this file
Uploaded                 : Takeout/Google Photos/Photos from 2020/VID_20200512_101540.mp4: VID_20200512_101540.mp4
Uploaded                 : Takeout/Google Photos/Photos from 2020/e6bcffd1178861ce50c2c2f755cb7f4b.mp4: e6bcffd1178861ce50c2c2f755cb7f4b.mp4
Managing albums
Scan of the sources:
   687 files in the input
--------------------------------------------------------
   339 photos
    23 videos
   325 metadata files
   108 files with metadata
     0 discarded files
     0 files having a type not supported
     0 discarded files because in folder failed videos
   687 input total (difference 0)
--------------------------------------------------------
   108 uploaded files on the server
     0 upgraded files on the server
     0 files already on the server
     0 discarded files because of options
     0 discarded files because duplicated in the input
     0 discarded files because server has a better image
     0 errors when uploading
   108 handled total (difference 254)
Done.

[simulot]

Have you processed all the parts of the takeout archive in the same run? It appears that images are spread across parts

[narucx]

Takeout made me download it in 32 different zips I thought the metadata would be in the same zip as the image/video file itself but ill try again after adding all files into a single zip

[simulot]

Just run immich-go on *.zip

[narucx]

yea it was split in 32 different zips

tried them 1 by 1 but that caused issues added all into one single zip and now i dont have a single error yet

[simulot]

As you want. But the simplest way was to do following:

./immich-go -server=http://mynas:2283 -key=thekey upload -google-photos   ~/Download/takeout-*.zip             

immich-go should open all of them, map the files and start upload. I haven't tried myself with that many files... I would like to know if this works

[narucx]

Ah yea that worked even better and uploaded the remaining files that previously had an error trying to upload them with 1 big folder. The ä got fucked up during copying ig (nothing to do with imich-go)

[simulot]

hum... I don't like the file names unshch,,refe3.png... Could you share the exact names of this files, the associated json and the json content?

[narucx]

try with the one big zip file immich-go.exe -log-file=logs3.txt -server=http://x.x -key=abc upload -create-albums -google-photos "Google Photos\takeout.zip"

try with the takeout-.zip with the 32 "small" zips `immich-go.exe -log-file=logs3.txt -server=http://x.x -key=abc upload -create-albums -google-photos "Google Photos\takeout-.zip"`

I tried both twice (deleted the images first so they'd be uploaded again) Big zip always fucks up the 'ä' and makes it to ',,' With the small zip files it doesn't do that

[simulot]

I think, I got it. For some reasons, the process for making the big file changes the encoding of the file names into something else than UTF-8.

According the zip documentation names in your big zip should be encoded using CP-437 (the one of IBM-PC for 1981!) or not:

    // NonUTF8 indicates that Name and Comment are not encoded in UTF-8.
        //
    // By specification, the only other encoding permitted should be CP-437,
    // but historically many ZIP readers interpret Name and Comment as whatever
    // the system's local character encoding happens to be.
    //
    // This flag should only be set if the user intends to encode a non-portable
    // ZIP file for a specific localized region. Otherwise, the Writer
    // automatically sets the ZIP format's UTF-8 flag for valid UTF-8 strings.

So I'll check how I can reproduce this case, and provide a fix.

Hopefully the zips generated by Google are encoded with UTF-8. So you have completed the upload, right?

[narucx]

Yea it worked for most files with the big zip and the rest went through with the small zips Everything is done for me now ty