search

boredazfcuk / docker-icloudpd

An Alpine Linux container for the iCloud Photos Downloader command line utility
1.7k stars 162 forks source link

iOS Shared Libraries #239

Closed timrettop closed 8 months ago

timrettop commented 1 year ago

Is there any suggestions on how to debug not seeing any SharedSync libraries when using docker exec -it icloudpd-shared /usr/bin/icloudpd --list-libraries with the icloudpd:testing docker container?

I'm looking to try out backing up a shared library to a separate backup location and am only seeing PrimarySync library.

I use iOS 16 and have shared a new album for testing (and have some that have been around for a while in my icloud library.

timrettop commented 1 year ago

Could it be that the PR here would fix that? https://github.com/icloud-photos-downloader/icloud_photos_downloader/pull/489

boredazfcuk commented 1 year ago

That’s the patch which I merge to enable shared library downloads. Without it, there is no --list-libraries command line option.

khargy commented 1 year ago

Not clear from the comments, does the latest version of the docker support Shared Libraries?

boredazfcuk commented 1 year ago

Not clear from the comments, does the latest version of the docker support Shared Libraries?

No, the version with the iOS16 shared library patches applied is: boredazfcuk/icloudpd:testing

I ran it for a while on my home server, but it broke after a couple of days. I don't even have shared libraries enabled on my account, so had to roll the code back to the previous version.

khargy commented 1 year ago

Not clear from the comments, does the latest version of the docker support Shared Libraries?

No, the version with the iOS16 shared library patches applied is: boredazfcuk/icloudpd:testing

I ran it for a while on my home server, but it broke after a couple of days. I don't even have shared libraries enabled on my account, so had to roll the code back to the previous version.

Thanks for the reply, I saw your note about the testing repository the other day. So far I haven't had any issues even with downloading from a large shared library.

epheterson commented 1 year ago

Did something change? Looks like this was working but stopped. I'm using icloudpd:latest (not testing, but have never used testing).

2022-12-12 13:16:30 INFO     iCloudPD launch command: /usr/bin/icloudpd --directory /home/(user)/iCloud --cookie-directory /config --folder-structure {:%Y/%m/%d} --username (email) --auto-delete --library SharedSync-(UUID)  2>/tmp/icloudpd/icloudpd_download_error
2022-12-12 13:16:31 ERROR    Failed to download new files
2022-12-12 13:16:31 ERROR     - Can you log into iCloud.com without receiving pop-up notifications?
2022-12-12 13:16:31 ERROR    Error debugging info:
2022-12-12 13:16:31 ERROR    Error: no such option: --library
2022-12-12 13:16:31 ERROR    ***** Please report problems here: https://github.com/boredazfcuk/docker-icloudpd/issues *****
boredazfcuk commented 1 year ago

Did something change? Looks like this was working but stopped. I'm using icloudpd:latest (not testing, but have never used testing).

The iOS shared library patches broke basic functionality for a lot of people (myself included): https://github.com/boredazfcuk/docker-icloudpd/issues/235

So I pulled the patches and added them to the testing container, as it seems that something isn't quite right with them.

epheterson commented 1 year ago

Interesting, I have two other personal library containers and both have been working (save one odd issue). I might move my container to testing, or wait for the code to be fixed and pushed back to latest. Thanks for the update!

epheterson commented 1 year ago

The odd issue I'm seeing is that some files will be downloaded then deleted on the same run, repeatedly. New files do seem to get added, so haven't really seen an issue with it. This never happened before the shared library changes, so might be related. It's still happening on the current latest tag, though, so maybe something else. Let me know if I should file a unique issue.

e.g.

2022-12-12 13:10:27 INFO     Generating list of files in iCloud. This may take a long time if you have a large photo collection. Please be patient. Nothing is being downloaded at this time
/home/username/iCloud/2021/03/19/IMG_8867.jpeg
2022-12-12 13:21:24 INFO     Downloading /home/username/iCloud/2021/03/19/IMG_8867.jpeg
2022-12-12 13:34:11 INFO     Deleting /home/username/iCloud/2021/03/19/IMG_8867.jpeg!

Logs: iCloudPD-logs.txt

boredazfcuk commented 1 year ago

I get this a lot. Have done for a long while. It's usually because I have saved files from an external app, such as Signal or Telegram, deleted them, and then realised I wanted them back, so saved them again (instead of recovering them from 'Recently Deleted'). This means I have two files with the same name. One in Photos and one in recently deleted.

When the script runs, it sees there is a photo in the photo stream named IMG_1234.HEIC (or whatever) and sees it does not exist on the local filesystem, so downloads it. After it finishes downloading, it scans the "Recently Deleted" folder, sees a file in there called IMG_1234.HEIC, sees that it matches a file in the photo stream and so removes it.

I usually just find the matching files in 'Recently Deleted' and remove them permanently.

epheterson commented 1 year ago

Thanks, that explains it. I think in my case this is happening because I used the merge duplicates feature in Photos app.

A simple fix, I think, would be to only delete a file iff it's in recently deleted and the same filename was not detected in the original sync / photo stream as a download/existing photo. This is kind of derailing the point of this thread, I can file a new ticket if we think this is worth doing.

Also seems like this will resolve itself once the items expire from Recently Deleted.

timrettop commented 1 year ago

Seeing you close this reminded me that I could never get the shared libary UUID to show. I just tried again using this command with the :testing branch and get an API error, any thoughts:

/ # icloudpd --list-libraries iCloud username/email: ---------@icloud.com 2023-04-10 12:48:09 DEBUG Authenticating... iCloud Password: Traceback (most recent call last): File "/usr/bin/icloudpd", line 33, in sys.exit(load_entry_point('icloudpd==1.7.2', 'console_scripts', 'icloudpd')()) File "/usr/lib/python3.10/site-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/usr/lib/python3.10/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/usr/lib/python3.10/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3.10/site-packages/click/core.py", line 535, in invoke return callback(args, kwargs) File "/usr/lib/python3.10/site-packages/icloudpd-1.7.2-py3.10.egg/icloudpd/base.py", line 293, in main libraries_dict = icloud.photos.libraries File "/usr/lib/python3.10/site-packages/pyicloud_ipd/services/photos.py", line 275, in libraries libraries[zone_name] = PhotoLibrary( File "/usr/lib/python3.10/site-packages/pyicloud_ipd/services/photos.py", line 154, in init request = self.service.session.post( File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 635, in post return self.request("POST", url, data=data, json=json, kwargs) File "/usr/lib/python3.10/site-packages/pyicloud_ipd/base.py", line 100, in request self._raise_error(code, reason) File "/usr/lib/python3.10/site-packages/pyicloud_ipd/base.py", line 122, in _raise_error raise api_error pyicloud_ipd.exceptions.PyiCloudAPIResponseError: INTERNAL_ERROR (INTERNAL_ERROR)

boredazfcuk commented 1 year ago

The testing container was just something I put together from a PR that was created in the upstream package. It's still not been merged though. Extensive work has been done to my container to fix an issue relating to authentication in China that would prevent me from merging the patches that were in the testing branch, so that's kind of dead now.

If the feature is added upstream, then I'll accommodate it here, but until that's done, there's not much point in keeping this open.

timrettop commented 1 year ago

Ahh, the same error was found and reported in the upstream PR that was never merged. https://github.com/icloud-photos-downloader/pyicloud/pull/8#issuecomment-1453072803

boredazfcuk commented 1 year ago

The upstream project seemed like it had stalled for a while, but then a flood of updates came in January/February... It's what kind of got me re-interested in the project (as well as increased personal time to work on it).

It's slowed down somewhat over the past month, but hopefully they're still working on it.

jcbshw commented 1 year ago

FYI looks like Shared Library is working now. I never had success with the testing container previously but after seeing recent success in https://github.com/icloud-photos-downloader/pyicloud/pull/8 I gave it another try.

Only issue I encountered is needing to run "chmod 750 /iCloud" to fix permissions but that's unrelated to Shared Library support.

boredazfcuk commented 1 year ago

That is the original PR which I merged into the testing container. It won't work on the latest version of the container as it's based on old code. If the PR gets rebased and merged upstream, I'll update the version bundled in the container and re-enable the functions. Until that happens though, the functionality will just have to stay in the testing container.

I will re-open this as a reminder though.

tangyue1 commented 1 year ago

I would love to see the upstream code works with shared library. I noticed that only my personal library is being downlaoded now.

laurensguijt commented 1 year ago

Is there any progress with the shared librarys? I would love to use this for a specific shared library.

lonevvolf commented 1 year ago

Looks like there actually is - check here: https://github.com/icloud-photos-downloader/icloud_photos_downloader/pull/678

tangyue1 commented 1 year ago

Really appreciate the hard work you put in. would like to see this feature added. For once it worked with the testing image but some update broke it.

epheterson commented 1 year ago

Somehow testing image is still working for me 🤷‍♂️ Also appreciate all the hard work and look forward to support in the main image someday!

jcbshw commented 11 months ago

Shared Library support was just merged upstream

https://github.com/icloud-photos-downloader/icloud_photos_downloader/pull/678

boredazfcuk commented 11 months ago

Looks like it's broken things at the minute though: https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/690

boredazfcuk commented 11 months ago

I've re-enabled the code which should support shared libraries again. I don't actually have any to test with though, so feedback would be appreciated.

timrettop commented 11 months ago

Looks like they're testing the bug in your link above, I get the same outcome with your latest update. I guess if they rev the version with a fix, you'll have to do the same?

boredazfcuk commented 11 months ago

Yeah, once they've pushed a fixed version, I'll just rebuild mine again.

Thought I'd push the current code out so that people using my container can report their findings. With more users testing, it will help ascertain whether the issue is widespread or only affecting a small number of users.

jcbshw commented 11 months ago

I can confirm downloading my shared library with the updated container is working.

I did not test --list-libraries since I already had my shared library name.

theskyisthelimit commented 11 months ago

I was looking forward for this feature. I tried running the --list-libraries but get the following message:

/ # sync-icloud.sh --list-libraries

2023-09-27 07:17:15 INFO     ***** boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.635 started *****
2023-09-27 07:17:15 INFO     ***** For support, please go here: https://github.com/boredazfcuk/docker-icloudpd *****
2023-09-27 07:17:15 INFO     Alpine Linux 3.18.3
2023-09-27 07:17:15 INFO     Python version: 3.11.5
2023-09-27 07:17:15 INFO     Loading configuration from: /config/icloudpd.conf
2023-09-27 07:17:18 INFO     Apple ID: XXXX
2023-09-27 07:17:18 INFO     Authentication Type: MFA
2023-09-27 07:17:18 INFO     Cookie path: /config/XXXX
2023-09-27 07:17:18 INFO     Cookie expiry notification period: 7
2023-09-27 07:17:18 INFO     Download destination directory: /home/user/iCloud
2023-09-27 07:17:18 INFO     Folder structure: {:%Y/%m/%d}
2023-09-27 07:17:18 INFO     Synchronisation interval: 86400
2023-09-27 07:17:18 INFO     Synchronisation delay (minutes): 0
2023-09-27 07:17:18 INFO     Set EXIF date/time: false
2023-09-27 07:17:18 INFO     Auto delete: false
2023-09-27 07:17:18 INFO     Delete after download: false
2023-09-27 07:17:18 INFO     Photo size: original
2023-09-27 07:17:18 INFO     Single pass mode: false
2023-09-27 07:17:18 INFO     Skip download check: false
2023-09-27 07:17:18 INFO     Skip live photos: false
2023-09-27 07:17:18 INFO     Number of most recently added photos to download: Download All Photos
2023-09-27 07:17:18 INFO     Downloading photos from album: Download All Photos
2023-09-27 07:17:18 INFO     Stop downloading when prexisiting files count is: Download All Photos
2023-09-27 07:17:18 INFO     Live photo size: original
2023-09-27 07:17:18 INFO     Skip videos: false
2023-09-27 07:17:18 INFO     Convert HEIC to JPEG: false
2023-09-27 07:17:18 INFO     Downloading from: icloud.com
2023-09-27 07:17:18 INFO     Authentication domain: com
2023-09-27 07:17:18 INFO     Ignore Synology extended attribute directories: Disabled
2023-09-27 07:17:18 WARNING  Ignoring invalid launch parameter specified: --list-libraries
2023-09-27 07:17:18 WARNING  Please do not specify the above parameter when launching the container. Continuing in 2 minutes

Am I doing something wrong?

ryan-adler commented 11 months ago

Same issue here - I'm also seeing #237 when trying to do a normal sync.

lonevvolf commented 11 months ago

Could it be that the --list-albums and --list-libraries params are missing here? https://github.com/boredazfcuk/docker-icloudpd/blob/60997762aa2fc5dba5022f066d2bb319706d8d8d/sync-icloud.sh#L1916

boredazfcuk commented 11 months ago

Could it be that the --list-albums and --list-libraries params are missing here?

https://github.com/boredazfcuk/docker-icloudpd/blob/60997762aa2fc5dba5022f066d2bb319706d8d8d/sync-icloud.sh#L1916

Yeah... I thought I'd just commented out all the stuff for shared libraries when I disabled it. Looks like I removed the command line parameter for that, and also for --photo-album too.

Both have been re-enabled. Codebase has moved to 1.16.1 too, so it will have the fix in it, which causes an error in certain circumstances.

theskyisthelimit commented 11 months ago

One step further but the script stops without listing the libraries.

2023-09-28 06:51:58 INFO     Directory is writable: /config/python_keyring/
2023-09-28 06:51:58 INFO     Directory is writable: /home/user/.local/share/
2023-09-28 06:51:58 INFO     Shared libraries available:
2023-09-28 06:51:59 DEBUG    Authenticating...
2023-09-28 06:52:03 INFO     Albums available:
2023-09-28 06:52:03 INFO      - PrimarySync
/ #
timrettop commented 11 months ago

It works for me! 🕺 Thank you!!!

@theskyisthelimit you might not have setup a shared library. I got confused with shared albums and shared library.

Check in your iOS Settings -> Photos and see if you've setup a Shared Album.

theskyisthelimit commented 11 months ago

It works for me! 🕺 Thank you!!!

@theskyisthelimit you might not have setup a shared library. I got confused with shared albums and shared library.

Check in your iOS Settings -> Photos and see if you've setup a Shared Album.

I have set up shared libraries.

boredazfcuk commented 11 months ago

It works for me! 🕺 Thank you!!!

I'm not sure how. I've looked at the code and could not see anywhere that a shared library would be downloaded. Yes, it had the capability to list the libraries by name and to list/download albums, but I couldn't see anywhere it would actually download the library.

I've had to give it quite a major reworking... basically copy/pasted a chunk of about 30 lines of code that downloads albums and convert it to run against libraries instead.

Hopefully it should work now. @theskyisthelimit can you pull the latest version and see what happens?

theskyisthelimit commented 11 months ago

Hopefully it should work now. @theskyisthelimit can you pull the latest version and see what happens?

@boredazfcuk , Much appreciated! Unfortunately, the behavior is still the same:

excerpt from the log: 

2023-09-28 23:15:20 INFO     ***** boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.643 started *****
.......
.......
2023-09-28 23:15:22 INFO     Shared libraries available:
2023-09-28 23:15:24 DEBUG    Authenticating...
2023-09-28 23:15:28 INFO     Libraries available:
2023-09-28 23:15:28 INFO      - PrimarySync
boredazfcuk commented 11 months ago

It is returning the name of a shared library though: PrimarySync

Have you tried setting photo_libary=PrimarySync and checked what it does/downloads?

fver commented 11 months ago

Looking into this, it seems you only get access to the shared library from the account that owns it, not from the account that receive the share. Can you try a list-library with the email and passwd from the owner of the share?

lonevvolf commented 11 months ago

The version that was current this morning is working for me - currently downloading my huge library since this morning. Seems to be some major confusion here...

boredazfcuk commented 11 months ago

I've found the bit that did the downloading on the version earlier today:

   if [ -z "${photo_album}" ] && [ "${photo_library}" ]; then
      command_line="${command_line} --folder-structure ${folder_structure}"
   fi
   if [ "${photo_album}" ]; then
      command_line="${command_line} --album ${photo_album}"
   elif [ "${photo_library}" ]; then
      command_line="${command_line} --library ${photo_library}"
   fi

I think this bit was a typo [ -z "${photo_album}" ] && [ "${photo_library}" ] and should have been [ -z "${photo_album}" ] && [ -z "${photo_library}" ]. That way, if neither an album or library is configured, it falls back to using the folder structure.

The following bit then adds the library to download to the command line:

   if [ "${photo_album}" ]; then
      command_line="${command_line} --album ${photo_album}"
   elif [ "${photo_library}" ]; then
      command_line="${command_line} --library ${photo_library}"
   fi

So that's where it was. I've tidied all that up now, so hopefully should be cleaner.

theskyisthelimit commented 11 months ago

It is returning the name of a shared library though: PrimarySync

Have you tried setting photo_libary=PrimarySync and checked what it does/downloads?

Tried it now but nothing "more" happens. My "primary" library, not the shared one is working fine. Im also the owner of the shared library.

I would expect to have 2 libraries here with the command --list-libraries

2023-09-28 23:15:28 INFO     Libraries available:
2023-09-28 23:15:28 INFO      - PrimarySync
???
theskyisthelimit commented 11 months ago

I let it run for a while and it created a new folder called PrimarySync in the same folder as the yearly folders are and downloaded all images from my primary library. But still nothing from the shared library. Is there like a default name for the shared library?

tangyue1 commented 11 months ago

The latest image downloads the shared folder but did not respect the folder_structure option. It downloads into a flat folder called "SharedSync-xxx"

theskyisthelimit commented 11 months ago

# sync-icloud.sh --list-libraries still not listing the shared library.

2023-10-01 05:53:26 INFO     ***** boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.649 started *****
........
2023-10-01 05:53:38 INFO     Shared libraries available:
2023-10-01 05:53:38 INFO      - PrimarySync
/ #

anything else I could try?

KUMAKAIHA commented 11 months ago

sync-icloud.sh --list-libraries

I also have the same problem. Only PrimarySync is displayed, and if photo_libary=PrimarySync is used, a PrimarySync folder will be generated, and unordered photos that are not in the shared gallery will be synchronized to this folder

yieldhog commented 11 months ago

The latest image downloads the shared folder but did not respect the folder_structure option. It downloads into a flat folder called "SharedSync-xxx"

I have the same issue -- Additionally, when I set the configuration "photo_library" to SharedSync-XXX it quit downloading my original non-shared library. When I set the variable to "all albums" I get the following error in logs:

2023-10-01 09:03:36 INFO Check successful 2023-10-01 09:03:36 INFO New files detected: 62 2023-10-01 09:03:38 INFO Downloading library: all albums Usage: icloudpd Try 'icloudpd -h' for help.

Error: Got unexpected extra arguments (albums albums albums)

Anyone find a solution to this?

boredazfcuk commented 11 months ago

Downloading library: all albums

Albums and Libraries are different concepts as far as iCloud is concerned. Attempting to download from a shared library named "all albums" doesn't sound like something that should work.

yieldhog commented 11 months ago

Downloading library: all albums

Albums and Libraries are different concepts as far as iCloud is concerned. Attempting to download from a shared library named "all albums" doesn't sound like something that should work.

I suppose that's true -- should probably update the configuration documentation.

photo_library: Set this to a comma delimited field to download photos from a shared library. Please note, if downloading from multiple libraries, you need to enclose them in quotes in your /config/icloudpd.conf file e.g. photo_library="one,two,three and four" will download photos from three libraries named "one", "two" and "three and four". When downloading photo libraries, the folder structure will be set to be the name of the library eg "/home/boredazfcuk/iCloud/one/IMG_0001.HEIC", "/home/boredazfcuk/iCloud/two/IMG_0002.HEIC" and "/home/boredazfcuk/iCloud/three and four/IMG_0003.HEIC". Set photo_library="all albums" in your configuration file /config/icloudpd.conf to download all albums. Please note: Due to a limitation in an upstream package, downloading from multiple libraries will trigger multiple download runs, and Apple may force a multi-factor re-authentication.

KUMAKAIHA commented 11 months ago

I think we need a quick solution to synchronize shared library photos from iOS16. Because once your photos are marked as a shared library, whether from yourself or others, they will not be synchronized to NAS in the existing version as usual. It is now known that using photo_library="all albums" will result in errors, and using photo_libary=PrimarySync will not result in the correct synchronization library