iOS 16 Shared Library Support

epheterson commented 2 years ago

Summary

In iOS 16 a new iCloud Photos Library feature was released called Shared Library which allows multiple iCloud users to combine photos in a common library, alongside their own personal libraries. Photos in this shared library currently are not backed up by iCloud photos downloader.

Could we please add support to download these photos?

Some considerations:

There should be a yes/no toggle to download these photos for each account (e.g. if multiple accounts are backed up, we don't want to duplicate).
Default should be yes to download these photos (e.g. for just one user backing up their own photos, these should be included for a complete backup).
Moving photos between personal/shared libraries should be supported. This can either delete/re-download, or switch context of the existing files. Photos app allows moving the photos between, so some consideration/testing should be done to make sure this works properly.

Context

I'm on the iOS 16 beta with another user and we're enjoying this feature, but it's not great that our shared library photos are no longer backed up.

Check out this article for some context.

epheterson commented 2 years ago

Looks like this update will be dependent on pyicloud adding support here: https://github.com/picklepete/pyicloud/issues/395

pneff commented 1 year ago

Done here: icloud-photos-downloader/icloud_photos_downloader#489

khargy commented 1 year ago

So it's not clear, is Shared Library Support working?

epheterson commented 1 year ago

Still working for me, but only in the Testing container.

Haven't tried switching back to latest tag. Initially it worked there, but then was removed because people saw sync failures with normal libraries. The same changes were moved to a Testing tag, and it continues to work fine there. Eventually I imagine things will be fixed and merged to latest, not sure when.

Edit: Sorry I'm referring to Testing tag on iCloudPD. Since that project is dependent on this one, maybe it is merged and ready to go in this project?

khargy commented 1 year ago

Still working for me, but only in the Testing container.

Haven't tried switching back to latest tag. Initially it worked there, but then was removed because people saw sync failures with normal libraries. The same changes were moved to a Testing tag, and it continues to work fine there. Eventually I imagine things will be fixed and merged to latest, not sure when.

Edit: Sorry I'm referring to Testing tag on iCloudPD. Since that project is dependent on this one, maybe it is merged and ready to go in this project?

Yes I wrote a separate ticket about the testing branch not working for me on the docker container all of a sudden a few days ago.

In the meantime I tried to setup things directly outside of docker and when I run it using the library option I get "Error: No such option: --library Did you mean --directory?". Is there a separate branch I need or is that the wrong option?

epheterson commented 1 year ago

Looks like this is tracking it and the pyicloud dependency isn't yet merged on, I'm not actually show how it ever worked yet: https://github.com/icloud-photos-downloader/icloud_photos_downloader/pull/489

txwireless commented 1 year ago

I haven't been able to get it to work. I've studied the code changes and what actually happens when you go to iCloud.com. It looks like a partial solution that possibly? might work for some but not consistently.

So what I noted was there is a 1-to-1 relationship between an iCloud account and ability to access a shared library (regardless of whether you are the sharer or sharee). The code points to the private endpoint which is where your non-shared library lives. One of the calls which gives photo counts, etc. also returns the GUID identifier of the shared library and the associated owner GUID. To access the actually shared library it looks like you have to hit the shared endpoint and provide both the shared library info and owner info to get the list of photos and download them. I confirmed this from both the sharer and sharee iCloud accounts that the same information is returned. I just don't have the time to figure out how to fix the code. The other caveat as noted elsewhere is the new advanced data protection basically breaks everything. While the website allows you an hour override, it doesn't seem very feasible to create a good solution. So shared library support will depend on NOT having advanced data protection enabled.

Specifics for getting shared library info are: Post request to {HOST}/database/1/com.apple.photos.cloud/production/private/records/lookup returns JSON response object with following key fields:

linkedShareZoneNameList - Type String_List - Value "SharedSync-{GUID}"
linkedShareZoneOwnerList - Value includes "recordName" equal to shared owner GUID

There is one other linkedShare value returned which I'm not sure how it is used or if needed:

linkedShareZoneJoinContinuationToken - Type Bytes_List - Value - random characters

Also, this request to private endpoint defines ZoneId as follows: "zoneID":{"zoneName":"PrimarySync","ownerRecordName":"{iCloud User GUID}","zoneType":"REGULAR_CUSTOM_ZONE"}

Then the same call is made but now to the shared endpoint: Post request to {HOST}/database/1/com.apple.photos.cloud/production/shared/records/lookup with one key distinction which is request json looks like: "zoneID":{"zoneName":"SharedSync-{GUID}","ownerRecordName":"{Shared Owner GUID}","zoneType":"REGULAR_CUSTOM_ZONE"}

The photo listing requests to shared also include the "zoneID" above containing both the SharedSync-{GUID} and {Shared Owner GUID}" for each call. Those calls look otherwise similar to the private endpoint calls, a sample would look like: {HOST}/database/1/com.apple.photos.cloud/production/shared/records/query

Hope this help someone dive down the rabbit hole further. Seems to me, instead of list libraries as an option, the user would want to specify at the cli whether they want personal/private, shared or both then the code would execute the various iterations accordingly.

vamitrou commented 1 year ago

it's been a while and no updates. Is it working now?

gshub77 commented 1 year ago

Bump? this is very important with ios 16 as I have shared albums for most of my favorite photos.

lonevvolf commented 1 year ago

Any chance of this making it into the released package?

AndreyNikiforov commented 1 year ago

Any chance of this making it into the released package?

IIUC there is no completed work for shared libraries support that is compatible with current the code of icloudpd (and have all existing functionality working proved by tests).

lonevvolf commented 1 year ago

It's a bit hard to follow what has been done where and what's actually working. It's unfortunate, especially since I've moved all of my photos into a Shared Library. I'm sure many others have as well. I wish I had time to work on this to help out but time is tight lately and I would have to invest a bit understanding the library before being able to attempt implementing anything. It's also not clear to me where the main challenge lies in implementing this yet.

AndreyNikiforov commented 1 year ago

It's a bit hard to follow what has been done where and what's actually working. It's unfortunate, especially since I've moved all of my photos into a Shared Library. I'm sure many others have as well. I wish I had time to work on this to help out but time is tight lately and I would have to invest a bit understanding the library before being able to attempt implementing anything. It's also not clear to me where the main challenge lies in implementing this yet.

icloudpd uses a fork of pyicloud. Original pyicloud lib was upgraded but the fork was neither synchronized nor icloudpd was switched to original pyicloud (multiple attempts exist on github but none has all features of icloudpd proved working through tests). IIUC, shared lib support was done to the fork of original pyicloud, so to use it, icloudpd has to be migrated to original pyicloud first. I cannot commit such amount of time yet.

txwireless commented 1 year ago

So I was never able to get this to work but I did find another project based on node that allows for photo downloading. https://github.com/steilerDev/icloud-photos-sync/wiki

To download shared libraries you have to do it from the master account and not one that has joined the shared library at this point. I ran into some issues with a larger library (discussed here https://github.com/steilerDev/icloud-photos-sync/issues/278) that included needing to manually apply a PR and up open file limits in the docker-compose.yml file but the net is it worked. I wouldn't call it perfect but I know I at least have another copy of my library now.

AndreyNikiforov commented 1 year ago

So I was never able to get this to work but I did find another project based on node that allows for photo downloading. https://github.com/steilerDev/icloud-photos-sync/wiki

To download shared libraries you have to do it from the master account and not one that has joined the shared library at this point. I ran into some issues with a larger library (discussed here steilerDev/icloud-photos-sync#278) that included needing to manually apply a PR and up open file limits in the docker-compose.yml file but the net is it worked. I wouldn't call it perfect but I know I at least have another copy of my library now.

Thanks for sharing. I hope we can get shared libs working for icloudpd project too. Maybe I need to start using shared libs for my family to motivate myself to commit more time to fixing icloudpd ;-)

lonevvolf commented 1 year ago

Maybe I need to start using shared libs for my family to motivate myself to commit more time to fixing icloudpd ;-)

This is the way. :)

tangyue1 commented 1 year ago

Hope that downloading shared library can be added soon. Thank you for all the work!

gklka commented 1 year ago

I am waiting this feature since Shared libraries were introduced. I migrated to them in December, and since then I have basically no backup, so I am living in fear :D

teh-hippo commented 1 year ago

Agree with all the above. Would love this feature. Not being able to sync photos properly is pretty worrying and this tool w/ shared library support would be awesome.

icloud-photos-downloader / icloud_photos_downloader

iOS 16 Shared Library Support #455

Summary

Context