iOS Shared Libraries

[timrettop]

Is there any suggestions on how to debug not seeing any SharedSync libraries when using docker exec -it icloudpd-shared /usr/bin/icloudpd --list-libraries with the icloudpd:testing docker container?

I'm looking to try out backing up a shared library to a separate backup location and am only seeing PrimarySync library.

I use iOS 16 and have shared a new album for testing (and have some that have been around for a while in my icloud library.

[jcbshw]

I think I figured it out. It looks like "sed '1d'" is cutting off the shared library name.

https://github.com/boredazfcuk/docker-icloudpd/blob/a5eda0f74f74d5ea9abe81e2b7b7d1f32d743e83/sync-icloud.sh#L697

Running the command manually inside the container without sed provides the shared library name.

libraries_with_dates is also commented out in sync-icloud.sh preventing folder structure to be followed

https://github.com/boredazfcuk/docker-icloudpd/blob/a5eda0f74f74d5ea9abe81e2b7b7d1f32d743e83/sync-icloud.sh#L33

[boredazfcuk]

Downloading library: all albums

Albums and Libraries are different concepts as far as iCloud is concerned. Attempting to download from a shared library named "all albums" doesn't sound like something that should work.

I suppose that's true -- should probably update the configuration documentation.

photo_library: Set this to a comma delimited field to download photos from a shared library. Please note, if downloading from multiple libraries, you need to enclose them in quotes in your /config/icloudpd.conf file e.g. photo_library="one,two,three and four" will download photos from three libraries named "one", "two" and "three and four". When downloading photo libraries, the folder structure will be set to be the name of the library eg "/home/boredazfcuk/iCloud/one/IMG_0001.HEIC", "/home/boredazfcuk/iCloud/two/IMG_0002.HEIC" and "/home/boredazfcuk/iCloud/three and four/IMG_0003.HEIC". Set photo_library="all albums" in your configuration file /config/icloudpd.conf to download all albums. Please note: Due to a limitation in an upstream package, downloading from multiple libraries will trigger multiple download runs, and Apple may force a multi-factor re-authentication.

I just copy/pasted that text and changed the word album to library; looks like I missed a couple. I've not actually commissioned the multiple libraries stuff either yet, which is why some lines of code are commented out. I don't use shared libraries so I've no idea how they work, how they're named (as far as the system is concerned). etc.

[boredazfcuk]

I think I figured it out. It looks like "sed '1d'" is cutting off the shared library name.

1d deletes the first line of text, which look like this:

2023-10-02 19:21:20 DEBUG    Authenticating...

Running the command manually inside the container without sed provides the shared library name.

When I run it inside the container without the sed command, I get:

/ # /opt/icloudpd_latest/bin/icloudpd --username blah@email.com --cookie-directory ${config_dir} --domain com --directory /dev/null --list-libraries
2023-10-02 20:11:54 DEBUG    Authenticating...
PrimarySync

which is to be expected, as I don't use libraries. Running it with sed produces the library list in a format which can be processed:

/ # /opt/icloudpd_latest/bin/icloudpd --username blah@e-mail.com --cookie-directory ${config_dir} --domain com --directory /dev/null --list-libraries | sed '1d'
PrimarySync

libraries_with_dates is also commented out in sync-icloud.sh preventing folder structure to be followed

I've not implemented this yet. I don't use shared libraries, so wasn't sure if multiple shared libraries was a thing, which is why it was commented out.

[jcbshw]

This is what happens for myself:

/ # /opt/icloudpd_latest/bin/icloudpd --username example@example.com --cookie-directory ${config_dir} --domain com --directory /dev/null --list-libraries
2023-10-02 12:22:54 DEBUG    Authenticating...
SharedSync-########-####-####-####-############
PrimarySync

/ # /opt/icloudpd_latest/bin/icloudpd --username example@example.com  --cookie-directory ${config_dir} --domain com --directory /dev/null --list-libraries | sed '1d'
2023-10-02 12:28:47 DEBUG    Authenticating...
PrimarySync

libraries_with_dates is also commented out in sync-icloud.sh preventing folder structure to be followed

I've not implemented this yet. I don't use shared libraries, so wasn't sure if multiple shared libraries was a thing, which is why it was commented out.

Users with shared libraries would just have PrimarySync and SharedSync. It would be nice to have that uncommented so we can have the folder structure followed for the shared library.

[theskyisthelimit]

I just ran # sync-icloud.sh --list-libraries | sed '1d' inside the docker container and I also obtained the SharedSync Library Name in the same format SharedSync-########-####-####-####-############

[theskyisthelimit]

I am able to download the shared library when adding it to the config like this:

photo_library=SharedSync-########-####-####-####-############

It then downloads the files into a seperate folder without the structure.

Tomorrow I will try both libraries, comma seperated in the config. Then the last thing missing would be (maybe as an option) to have all files from both libraries in the same folder structure (folder_structure={:%Y/%m/%d}).

[boredazfcuk]

This is what happens for myself:

/ # /opt/icloudpd_latest/bin/icloudpd --username example@example.com --cookie-directory ${config_dir} --domain com --directory /dev/null --list-libraries
2023-10-02 12:22:54 DEBUG    Authenticating...
SharedSync-########-####-####-####-############
PrimarySync

/ # /opt/icloudpd_latest/bin/icloudpd --username example@example.com  --cookie-directory ${config_dir} --domain com --directory /dev/null --list-libraries | sed '1d'
2023-10-02 12:28:47 DEBUG    Authenticating...
PrimarySync

libraries_with_dates is also commented out in sync-icloud.sh preventing folder structure to be followed

I've not implemented this yet. I don't use shared libraries, so wasn't sure if multiple shared libraries was a thing, which is why it was commented out.

Users with shared libraries would just have PrimarySync and SharedSync. It would be nice to have that uncommented so we can have the folder structure followed for the shared library.

Are you running an older version of the container? There was an upstream change which pushed info out to stderr instead of stdout. That could be what's happening... The DEBUG line may be getting pushed to stderr, so displayed in console, but ignored by sed.

I've pushed a new version which enables the configuration items for shared libraries with folder structure. Please test and let me know.

[boredazfcuk]

Tomorrow I will try both libraries, comma seperated in the config. Then the last thing missing would be (maybe as an option) to have all files from both libraries in the same folder structure (folder_structure={:%Y/%m/%d}).

Folder structure wasn't implemented for Shared Libraries. It is in the version I've just pushed now though.

[tangyue1]

I set the photo_library=SharedSync-xxx and folder structure is still not respected in the latest image. Can it NOT add the SharedSync-xxx folder and just create folder structure under the set volume?

[boredazfcuk]

Can it NOT add the SharedSync-xxx folder and just create folder structure under the set volume?

No, people wanting to download from multiple shared libraries would have their libraries merged.

If you want to have libraries with dates, you need to set libraries_with_dates=true, same as how downloading from photo albums is handled.

[tangyue1]

libraries_with_dates=True, photo_library=SharedSync-xxx, Stop condition is either until_found=100 or recent_only=100 The latest image won't download any photo. 2023-10-04 09:22:22 2023-10-04 09:22:22 INFO Downloading library: SharedSync-xxx 2023-10-04 09:23:50 2023-10-04 09:23:50 INFO Synchronisation complete for user

[boredazfcuk]

If you remove the stop conditions, does work?

I'm wondering if the new code to download the photos doesn't consider these settings.

[tangyue1]

If you remove the stop conditions, does work?

I'm wondering if the new code to download the photos doesn't consider these settings.

I removed the stop condition but it still does not download. Here is the debugging, "--library" seems to be invoked twice.

2023-10-04 17:45:11 2023-10-04 17:45:11 INFO Downloading library: SharedSync-xxx 2023-10-04 17:45:11 2023-10-04 17:45:11 DEBUG iCloudPD launch command: /opt/icloudpd_latest/bin/icloudpd --directory /home/user/iCloud --cookie-directory /config --domain com --username xxx --no-progress-bar --library SharedSync-xxx --folder-structure SharedSync-xxx/{:%Y/%Y-%m/%Y-%m-%d} --library SharedSync-xxx 2>/tmp/icloudpd/icloudpd_download_error 2023-10-04 17:45:11 2023-10-04 17:45:11 DEBUG Set owner on iCloud directory, if required 2023-10-04 17:45:31 2023-10-04 17:45:31 DEBUG Set group on iCloud directory, if required 2023-10-04 17:45:50 2023-10-04 17:45:50 DEBUG Set 750 permissions on iCloud directories, if required 2023-10-04 17:46:09 2023-10-04 17:46:09 DEBUG Set 640 permissions on iCloud files, if required 2023-10-04 17:46:28 2023-10-04 17:46:28 INFO Synchronisation complete for user

[boredazfcuk]

Can you update to latest version and test please?

[tangyue1]

It is working now. Listing individual libraries like "PrimarySync,SharedSync-########-####-####-####-############" works. But photo_library="all libraries" does not work.

[lonevvolf]

I am also not seeing the correct folder structure under the SharedSync-### folder in the newest image. It's still flat for me.

[boredazfcuk]

I am also not seeing the correct folder structure under the SharedSync-### folder in the newest image. It's still flat for me.

Did you configure it to download to a dated file structure?

[lonevvolf]

folder_structure={:%Y/%m/%d}

[boredazfcuk]

folder_structure={:%Y/%m/%d}

So no then? folder_structure is ignored by photo/library downloads, in favour of named folders, unless confgured otherwise.

[lonevvolf]

I guess I'm confused - the doc says that setting that value, or leaving it blank will download files into the date structure. How can I configure it to use the date structure then?

EDIT: Never mind - found the parameter you're referring to. It was incorrectly set.

[yieldhog]

Am I having the same issue? I've been using this for about a year or so .. all of my files were downloaded neatly into the default date folders: ie 2023 // 09 // 25

I added the following ENV to my docker container: photo_library: "PrimarySync,SharedSync-ECEXXX"

overnight, it redownloaded all of my 52,000 pictures into a new subfolder "PrimarySync" with no folder structure. What is the best option here? Thanks

[KUMAKAIHA]

Can it NOT add the SharedSync-xxx folder and just create folder structure under the set volume?

No, people wanting to download from multiple shared libraries would have their libraries merged.

If you want to have libraries with dates, you need to set libraries_with_dates=true, same as how downloading from photo albums is handled.

When I use 'libraries_with_dates=true' in Docker settings, whether I use Ture or true, it will still be set to false in the icloudpd.conf file

[KUMAKAIHA]

Can it NOT add the SharedSync-xxx folder and just create folder structure under the set volume?

No, people wanting to download from multiple shared libraries would have their libraries merged. If you want to have libraries with dates, you need to set , same as how downloading from photo albums is handled.libraries_with_dates=true

When I use 'libraries_with_dates=true' in Docker settings, whether I use Ture or true, it will still be set to false in the icloudpd.conf file

In Docker version 1.0.665. Once photo_library is used, as libraries_with_dates cannot be set to true, photos will not be synchronized according to the file structure of the date. Meanwhile, due to the invalid setting of photo_library="all libraries", only photo_library= "PrimarySync,SharedSync-XXX" is allowed to synchronize personal libraries simultaneously. And due to the use of libraries_with_dates=false, the personal library will not be able to expand according to the date structure. Currently, once synchronization of shared libraries begins, it will result in personal libraries being unable to be synchronized or being synchronized in the PrimarySync folder without expanding by date.

[boredazfcuk]

Maybe... I think it's not working because of the typo when it generates the conf file, it's something like: libraries_with_datess

I spotted it this morning but not near a computer for a while.

[KUMAKAIHA]

Maybe... I think it's not working because of the typo when it generates the conf file, it's something like: libraries_with_datess

I spotted it this morning but not near a computer for a while.

Like this ? # if [ "${libraries_with_dates}" ]; then sed -i "s%^librariess_with_dates=.*%librariess_with_dates=${librariess_with_dates}%" "${config_file}"; fi

[boredazfcuk]

Yeah that's it. Turns out I've got the coronavirus, so just trying to sleep through the worst of it. Maybe pick this up in a few days as I don't have my laptop to alter the code.

If any of you guys want to submit a PR with the change I can approve it and kick off the build/deployment from my phone. Otherwise you're just gonna have to wait a few days till I'm good again.

[KUMAKAIHA]

Yeah that's it. Turns out I've got the coronavirus, so just trying to sleep through the worst of it. Maybe pick this up in a few days as I don't have my laptop to alter the code.

If any of you guys want to submit a PR with the change I can approve it and kick off the build/deployment from my phone. Otherwise you're just gonna have to wait a few days till I'm good again.

Wishing you a speedy recovery. I don't know much about GitHub and Docker, but I deleted the extra s in line 108 'librariess_with_dates' and submitted a PR

[tangyue1]

Wish you a speedy recovery! feel better!

[boredazfcuk]

Definitely on the mend, feeling loads better in last few hours. Should be back to normal by tomorrow.

[boredazfcuk]

I've pushed another update today. I found a bug in how albums are processed and made quite a few changes to that.

The Libraries functionality was basically a copy/paste job, so a few of these bugs were in there too. Hopefully it should work a little better now.

[m4rt0s]

Hi Boredazfcuk, first of all thank you for your great work!

My question is about synchronisation when using a shared library, is there a way to make it download both libraries in one folder instead of using separate folders?

Under photos folder I have right now this structure:

|── PrimarySync │   ├── 2020 │   ├── 2021 │   ├── 2022 │   └── 2023 |── SharedSync-XXXX-XXX-XXXX-XXX-XXXXXX │   ├── 2020 │   ├── 2021 │   ├── 2022 │   └── 2023

I would like it to look like this:

|── Photos │   ├── 2020 │   ├── 2021 │   ├── 2022 │   └── 2023

Is there any possibility to merge the two folders into one?

Thanks in advance.

[boredazfcuk]

Hi,

Personally, I don't download from Photo Libraries, so I've no experience of how they behave... but I think you may run into issues attempting to merge the libraries like that.

I'd imagine it's entirely possible that two different photos exist in each library, but use the same name IMG_0000.HEIC, for example.

After the first file is downloaded from the PrimarySync, the second file from SharedSync will be de-duplicated, so named something like IMG_0000-12345.HEIC.

After this, say you decided to delete IMG_0000.HEIC from the SharedSync library, the app finds IMG_0000.HEIC in deleted items. It has no way of knowing which Library that file came from, so it would delete the file on disk named IMG_0000.HEIC, which would be the file from PrimarySync.

[boredazfcuk]

Resolved long ago... just doing some long overdue housekeeping.