JSON of asset without URL

jtagcat commented 3 months ago

Some of my takeouts have files like the following:

{
  "title": "IMG_5785.JPG",
  "description": "",
  "imageViews": "2833",
  "creationTime": {
    "timestamp": "1383742293",
    "formatted": "6. nov 2013 12:51:33 UTC"
  },
  "photoTakenTime": {
    "timestamp": "1383732375",
    "formatted": "6. nov 2013 10:06:15 UTC"
  },
  "geoData": {
    "latitude": 0.0,
    "longitude": 0.0,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "geoDataExif": {
    "latitude": 0.0,
    "longitude": 0.0,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  }
}

These are determined to be Unknown JSON files.

isAsset should be true, but it is done on the assumption of the (in-code) example JSON:

        "title": "20161018_140312.mp4",
        "description": "",
        "imageViews": "87",
        "creationTime": {
            "timestamp": "1476820554",
            "formatted": "18 oct. 2016, 19:55:54 UTC"
        },
        "photoTakenTime": {
            "timestamp": "1476792192",
            "formatted": "18 oct. 2016, 12:03:12 UTC"
        },
        "geoData": {
            "latitude": 12.345,
            "longitude": 1.12345,
            "altitude": 0.0,
            "latitudeSpan": 0.0,
            "longitudeSpan": 0.0
        },
        "geoDataExif": {
            "latitude": 12.345,
            "longitude": 1.12345,
            "altitude": 0.0,
            "latitudeSpan": 0.0,
            "longitudeSpan": 0.0
        },
        "url": "https://photos.google.com/photo/....",
        "googlePhotosOrigin": {
            "mobileUpload": {
            "deviceFolder": {
                "localFolderName": ""
            },
            "deviceType": "ANDROID_PHONE"
            }
        }
    }

..where url is present. I patched my code to first differentiate between the two error conditions, and then as a hotfix to assume unknown JSON is of assets.

In the non-verbose log JSON File not found for this file looks confusing. failed to associate JSON file would be more appropriate.

simulot commented 3 months ago

What is special with this photo? What is its origin?

jtagcat commented 3 months ago

Nothing special with the photo, might be an older takeout (2018, previously exported and deleted from gphotos). Before the patch I got 0 uploads with 7k files. I checked a few other files, same thing.

Takeout of an another account with 2012..2021 in the export got 513/-33517 uploaded, with the same issue and alike JSONs. Same patch → no problems.

simulot commented 3 months ago

The takeout can have various JSON for print works or some other things. Unknown JSON can't be taken as asset.

jgoclawski commented 3 weeks ago

It's a thing with older Takeouts, that the JSON has a different format. I tried to pull both new Takeout and an older one which had more data and hit the same problem that from the old takeout 0 files would be imported. Album JSON is also a little bit different, because it includes a top-level "albumData" key.

simulot commented 3 weeks ago

could you share some samples of photo and albums json ?

jgoclawski commented 3 weeks ago

@simulot Absolutely.

I'll post metadata JSON from the same album, but different takeouts. Old takeout is from April 2020, the new one is from June 2024. My locale is Polish, so some data is localized.

Old takeout - file metadane.json (album) :

{
  "albumData": {
    "title": "Trip to Gdańsk",
    "description": "",
    "access": "protected",
    "location": "",
    "date": {
      "timestamp": "1502439626",
      "formatted": "11 sie 2017, 08:20:26 UTC"
    },
    "geoData": {
      "latitude": 0.0,
      "longitude": 0.0,
      "altitude": 0.0,
      "latitudeSpan": 0.0,
      "longitudeSpan": 0.0
    }
  }
}

^ notice the additional top-level key albumData.

New takeout - file metadane.json (album):

{
  "title": "Trip to Gdańsk",
  "description": "",
  "access": "protected",
  "date": {
    "timestamp": "1502439626",
    "formatted": "11 sie 2017, 08:20:26 UTC"
  },
  "geoData": {
    "latitude": 0.0,
    "longitude": 0.0,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  }
}

Old takeout - file IMG_20170803_115431469_HDR.jpg.json (asset - photo):

{
  "title": "IMG_20170803_115431469_HDR.jpg",
  "description": "",
  "imageViews": "0",
  "creationTime": {
    "timestamp": "1502439626",
    "formatted": "11 sie 2017, 08:20:26 UTC"
  },
  "modificationTime": {
    "timestamp": "1585318092",
    "formatted": "27 mar 2020, 14:08:12 UTC"
  },
  "geoData": {
    "latitude": 54.51708608333333,
    "longitude": 18.54171638888889,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "geoDataExif": {
    "latitude": 54.51708608333333,
    "longitude": 18.54171638888889,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "photoTakenTime": {
    "timestamp": "1501754071",
    "formatted": "3 sie 2017, 09:54:31 UTC"
  }
}

New takeout - file IMG_20170803_115431469_HDR.jpg.json (asset - photo):

{
  "title": "IMG_20170803_115431469_HDR.jpg",
  "description": "",
  "imageViews": "15",
  "creationTime": {
    "timestamp": "1502439626",
    "formatted": "11 sie 2017, 08:20:26 UTC"
  },
  "photoTakenTime": {
    "timestamp": "1501754071",
    "formatted": "3 sie 2017, 09:54:31 UTC"
  },
  "geoData": {
    "latitude": 54.5170861,
    "longitude": 18.5417164,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "geoDataExif": {
    "latitude": 54.5170861,
    "longitude": 18.5417164,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "url": "https://photos.google.com/photo/AF1QipNp7f29ZWPIDWAPMXJcNB2z7EMAGXWeTT066p9H",
  "googlePhotosOrigin": {
    "mobileUpload": {
      "deviceFolder": {
        "localFolderName": ""
      },
      "deviceType": "ANDROID_PHONE"
    }
  }
}

simulot commented 3 weeks ago

Thanks, it definitively deserve a look

jtagcat commented 6 days ago

Pull is included in 0.18.0, tested working.

simulot / immich-go

JSON of asset without URL #212