Hash values in photo._master_record["fields"]["resOriginalRes"]["value"]

takeru commented 1 year ago

The problem

value = photo._master_record["fields"]["resOriginalRes"]["value"]

In the value to 'fileChecksum' 'wrappingKey' 'referenceChecksum' What is the meaning of these hash values? I want to check the checksum of a downloaded file. I wish I knew the algorithm and encoding, but does anyone know?

Environment

pyiCloud release with the issue (pip show pyicloud): 1.0.0
Last working pyiCloud release (if known):
Service causing this issue:
Python version (python -V): Python 3.10.8
Operating environment (project deps/Docker/Windows/etc.): Mac

Traceback/Error logs

Checklist

[x] I've looked informations into the README.
[x] I've looked informations into the pyiCloud's code.
[x] I've looked informations in Google.

Additional information

takeru commented 1 year ago

https://developer.apple.com/library/archive/documentation/DataManagement/Conceptual/CloudKitWebServicesReference/Types.html

Asset Dictionary

An asset dictionary represents an Asset field type with the following keys:

Key | Description -- | -- fileChecksum | The signature of a file returned from the file upload step. This key is required for the public and private database. size | The size of the file, calculated by the asset upload step. This key is required for the public and private database. referenceChecksum | The checksum of the wrapping key returned from the upload step. This key is required for the private database. wrappingKey | The secret key used to encrypt the asset. This key is required for the private database. receipt | The receipt for uploading the asset, described in Upload Asset Data. This key is required for the public and private database when saving the enclosing record. downloadURL | The location of the asset data to download. This key is present only when fetching the enclosing record.

Asset Dictionary An asset dictionary represents an Asset field type with the following keys: https://developer.apple.com/library/archive/documentation/DataManagement/Conceptual/CloudKitWebServicesReference/UploadAssets.html I don't know the details, it seems to be returned when uploaded.

takeru commented 1 year ago


sample_photos = []
for _photo in api.photos.all:
    sample_photos.append(_photo)
    if 10<=len(sample_photos):
        break

import base64

for index, photo in enumerate(sample_photos):
    # print(json.dumps(photo._master_record,indent=2))
    print(f"[{index}]")

    fileChecksum = photo._master_record["fields"]["resOriginalRes"]["value"]["fileChecksum"]
    downloadURL = photo._master_record["fields"]["resOriginalRes"]["value"]["downloadURL"]
    print("curl -o '%s_%s' '%s'" % (fileChecksum.replace("+","-").replace("/","_"), photo.filename, downloadURL))

    b64 = fileChecksum
    print(b64)
    bin = base64.b64decode(b64)
    print(bin)
    hx = bin.hex()
    print(hx)

    print(len(hx))
    print("")

When I decode base64 to hex, the first byte seems to be fixed at 01. Is this a version of the hash algorithm? Is it something like 20byte = 160bit algorithm?

landall commented 1 year ago


sample_photos = []
for _photo in api.photos.all:
    sample_photos.append(_photo)
    if 10<=len(sample_photos):
        break

import base64

for index, photo in enumerate(sample_photos):
    # print(json.dumps(photo._master_record,indent=2))
    print(f"[{index}]")

    fileChecksum = photo._master_record["fields"]["resOriginalRes"]["value"]["fileChecksum"]
    downloadURL = photo._master_record["fields"]["resOriginalRes"]["value"]["downloadURL"]
    print("curl -o '%s_%s' '%s'" % (fileChecksum.replace("+","-").replace("/","_"), photo.filename, downloadURL))

    b64 = fileChecksum
    print(b64)
    bin = base64.b64decode(b64)
    print(bin)
    hx = bin.hex()
    print(hx)

    print(len(hx))
    print("")

When I decode base64 to hex, the first byte seems to be fixed at 01. Is this a version of the hash algorithm? Is it something like 20byte = 160bit algorithm?

It seems the algorithm is not a public algorithm. You should only use the checksum but can not calculate it.

picklepete / pyicloud