QNAP - Container Station Configuration for docker-icloudpd (Help) ?

[nodecentral]

Hi

First many thanks for posting the time/effort and instructions you've done around this.

I've just created an instance on my QNAP NAS (via its container station app) and when I completed the configuration and did the build, it run for the first time and on the console, i did not see a

"ERROR    Apple ID password set to 'usekeyring' but keyring file does not exist. Container must be run interactively to add a password to the system keyring - Restart in 5mins"

... but it went on to prompted me with the following.

...
2021-01-18 12:21:25 WARNING  Using Apple ID password from variable. This password will be visible in the process list of the host. Please add your password to the system keyring instead                                                                                                                                                                                                                                
2021-01-18 12:21:40 INFO     Correct owner on config directory, if required                                                                                                                                                                                                                                                                                                                                              
2021-01-18 12:21:40 INFO     Correct group on config directory, if required 
2021-01-18 12:07:06 INFO     Generate 2FA cookie with password: mypassword
2021-01-18 12:07:06 WARNING  Checking for new files using insecure password method. Please store password in the iCloud keyring to prevent password leaks                                                                                                                                                                                                                                                                
  0: SMS to ********66                                                                                                                                                                                                                                                                                                                                                                                                   
  1: Enter two-factor authentication code                                                                                                                                                                                                                                                                                                                                                                                
Please choose an option: [0]: 273684                                                                                                                                                                                                                                                                                                                                                                                     
Error: 273684 is not in the valid range of 0 to 1.                                                                                                                                                                                                                                                                                                                                                                       
Please choose an option: [0]: 1                                                                                                                                                                                                                                                                                                                                                                                          
Please enter two-factor authentication code: 283684                                                                                                                                                                                                                                                                                                                                                                      
2021-01-18 12:08:42 INFO     Two factor authentication cookie generated. Sync should now be successful. 

Yet I've not run the sync-icloud.sh or built/run a seperate instance yet) as I had the authorisation code there on my phone so I entered it in and it suggests it was successful ?

Is this message correct or do I still need to run the sync-icloud.sh etc ?

[nodecentral]

Update 1,

As nothing looked like it was downloading, I restarted the container and this time I ran the sync-icloud.sh file seperately, and again it completed the authentication yet, I'm not seeing any images download.

FYI - Below are my variables and i can see a file and a folder in ./config (which is mapped folder on my NAS), but no photos have been downloaded to the /home/john/iCloud (also mapped folder on my NAS)? - see shared folder config below

Volume from host	Mount Point
/home/john/iCloud	/share/Backups/VM/icloudpd/photos	Read/Write
/config	/share/Backups/VM/icloudpd	Read/Write

[nodecentral]

Update 2.

While it's still not downloading anything, I can confirm thatI do have what looks like a cookie in the .config folder (folder contents are below)

/config

• photos
• python_keyring
• myidemailaddressfile
• myidemailaddressfile.bak

FYI - The photos folder (which I created during the build to store them all) and the python_keyring folder are both empty. The myidemailaddress file, however that is created, has content like the following (which I've edited)

#LWP-Cookies-2.0
Set-Cookie3: X-APPLE-WEBAUTH-HSA-TRUST-FNR6DWHICH IHAVE EDITIED SO DOES NOT WORKMETAVJQSFZUQS="\"v=1:t=DA==BST_IAAAAAfwAABLwIAAAAAGAFg-QRDmwfdzLmljbG91ZC5hdXRovQAqxy8C8oOq443WdBfgeRRmp-fT3wefZC1QxTOInvariousplacesmgZWQeRtRMiLjXuBjWupM0v-_NbJRstImKSykgnUPJewfwef342r23qwefigZ0TYNdce7Xbxch1CMci-W_aFxx4cEO4gwefwefqhvW1hi8cMBc8RAeSEq8SkxCg~~\""; path="/"; domain=".icloud.com"; path_spec; domain_dot; secure; expires="2021-04-18 12:49:40Z"; HttpOnly=None; version=0
Set-Cookie3: X_APPLE_WEB_KB-FNwfR6D71APDwefwfMETAVfJQSFZUQS="\"v=1:t=DA==BST_IAAfAAAAABLwIAAAAAGAFg9IRDwefwefwef1ZC5hdXRwovQCqZydiGDWeGWOn3ZMKWyNv3P93Wt_sthulgCGLV-ghO-xH6zi9yjH-bWtKQgjSH-_gIdBS_oJ_kFEgMwefwfwefweef4KWGdqqv-k3-b09jcn_L3eMqjU0A4FAaNYeXqMoEMfLqj-wefNMq4G-frTRh2mxx11gdFllOSymmg~~\""; path="/"; domain=".icloud.com"; path_spec; domain_dot; secure; expires="2021-03-19 12:49:22Z"; HttpOnly=None; version=0

[boredazfcuk]

What does your log say? If there is a problem downloading files, it should tell you what it is.

[nodecentral]

What does your log say?

Sorry, where would i find/access the log ?

[boredazfcuk]

Hi,

Unfortunately, I'm not familiar with the QNAP containerisation app... But the log should look like the code snippet you have in your first post. Is there not something in there which shows you what the container is doing?

I kick out pretty much everything the container does in the same format so it should give loads of info.

Thanks.

[nodecentral]

There's a "console" view associated with any container I create on my QNAP, and that's where I would expect to see any info/errors/warning messages etc.

To help - I've stopped and restarted the container, and I can see there are a number of INFO lines, with two WARNINGs (below)

2021-01-18 14:21:25 WARNING  Additional command line options is depreceated. Please specify all options using the dedicated variables: --folder-structure={:%Y/%m}
2021-01-18 14:21:25 WARNING  Using Apple ID password from variable. This password will be visible in the process list of the host. Please add your password to the system keyring instead 

but then at the end, there is nothing after this.

2021-01-18 14:21:40 INFO     Generate 2FA cookie with password: mypassword
2021-01-18 14:21:40 WARNING  Checking for new files using insecure password method. Please store password in the iCloud keyring to prevent password leaks                                                                                                                                                                                                                                                                
  0: SMS to ********66                                                                                                                                                                                                                                                                                                                                                                                                   
  1: Enter two-factor authentication code  

Do I have re-authenticate with a code after each restart of the container ? Should the python-keyfile folder be empty ?

[boredazfcuk]

I think this may be due to an issue I've seen on Synology NAS devices. They always run the container interactively, so it will always prompt for you to confirm the two factor authentication. Try setting the interactive_only variable to True and it will probably skip over the two factor authentication prompt and progress with the downloads.

Should the python-keyfile folder be empty ?

Yes, it will only generate a keyring file if you are using keyring authentication, however, I can see from the log snippit that you've specified your password in a variable instead.

[nodecentral]

Hi, I've updated the subuject to make this thread more relevent to QNAP, so not to confuse matters - I've also come accross another issue and that is the "Binds" - as i get this error message

2021-01-18 18:14:32 INFO     Check download directory mounted correctly                                                                                                                                                                                                                                                                                                                                                  
2021-01-18 18:14:32 ERROR    Failsafe file /home/user/iCloud/.mounted file is not present. Plese check the host's target volume is mounted - retry in 5 minutes     

Running the following i get this..

 [~] # docker inspect icloudpd-2 | grep -A 3 'Binds'
            "Binds": [
                "/share/Backups/VM/icloudpd/photos:/home/user/iCloud:rw",
                "/share/Backups/VM/icloudpd:/config:rw"

What's strange is that the /config folder is fine, and content is written there, but not to the photos folder I have mapped the home/user/iCloud to ?

I'm unable to create a file called "/home/${user}/iCloud/.mounted" in the /share/Backups/VM/icloudpd/photos as it has illegal charactors ?? *I'm obviously missing something very obvious here :-)

Any ideas ?

[nodecentral]

Sorry @boredazfcuk,

I'm getting stuck, trying a number of things, then going away and coming back to make progress - reading it again, again , I've now have created a file called ".mounted" and placed it /share/Backups/VM/icloudpd/photos on my NAS and we've not progressed further, but only to a point.

2021-01-18 18:39:32 ERROR    Failsafe file /home/user/iCloud/.mounted file is not present. Plese check the host's target volume is mounted - retry in 5 minutes                                                                                                                                                                                                                                                          
2021-01-18 18:44:32 INFO     Failsafe file /home/user/iCloud/.mounted exists                                                                                                                                                                                                                                                                                                                                             
2021-01-18 18:44:32 INFO     Set owner, user, on iCloud directory, if required                                                                                                                                                                                                                                                                                                                                           
2021-01-18 18:44:32 INFO     Set group, group, on iCloud directory, if required                                                                                                                                                                                                                                                                                                                                          
2021-01-18 18:44:32 INFO     Correct owner on icloudpd temp directory, if required                                                                                                                                                                                                                                                                                                                                       
2021-01-18 18:44:32 INFO     Correct group on icloudpd temp directory, if required                                                                                                                                                                                                                                                                                                                                       
2021-01-18 18:44:32 INFO     Correct owner on config directory, if required                                                                                                                                                                                                                                                                                                                                              
2021-01-18 18:44:32 INFO     Correct group on config directory, if required                                                                                                                                                                                                                                                                                                                                              
2021-01-18 18:44:32 INFO     Correct owner on keyring directory, if required                                                                                                                                                                                                                                                                                                                                             
2021-01-18 18:44:32 INFO     Correct group on keyring directory, if required                                                                                                                                                                                                                                                                                                                                             
2021-01-18 18:44:32 INFO     Set 750 permissions on iCloud directories, if required                                                                                                                                                                                                                                                                                                                                      
2021-01-18 18:44:32 INFO     Set 640 permissions on iCloud files, if required                                                                                                                                                                                                                                                                                                                                            
2021-01-18 18:44:32 INFO     Command line options: --directory /home/user/iCloud --cookie-directory /config --folder-structure {:%Y/%m/%d} --username myappleidemail --password ******                                                                                                                                                                                                                      
2021-01-18 18:44:32 INFO     Sync user user                                                                                                                                                                                                                                                                                                                                                                              
2021-01-18 18:44:32 INFO     Check 2FA Cookie                                                                                                                                                                                                                                                                                                                                                                            
2021-01-18 18:44:32 INFO     Cookie exists, check expiry date                                                                                                                                                                                                                                                                                                                                                            
2021-01-18 18:44:32 ERROR    Cookie is not 2FA capable, authentication type may have changed. Please run container interactively to generate - Retry in 5 minutes     

I tried running another instance using..

[~] # docker exec -it icloudpd-2 /usr/local/bin/sync-icloud.sh

But I get the same message, I can't seem to get the 2FA prompt options again, is that due to the environment value "interative_only" being "True" ?

[boredazfcuk]

Hi,

Each time I type a reply, you update your comment and I have to start again lol

Connect to the container's command line with:

docker exec -it icloudpd-2 /bin/ash

From the command line just run unset interactive_only and run the script again:

/usr/local/bin/sync-icloud.sh

You should then be able to re-do your 2FA authentication.

Once that's done, restart the container so that it re-sets the interactive_only variable to True

[nodecentral]

Each time I type a reply, you update your comment and I have to start again lol

Sorry :( - I’m just so keen to get this working :-)

That looks like it helped, as I can see loads of IMG_XXXX files hitting the console .. but nothing is going into the /share/Backups/VM/icloudpd/photos ?

Does it have to do something first, before it can download anything ?

[boredazfcuk]

When it runs, it displays the list of files on the server that do not exist on the local disk. Once it has finished displaying the new file list, it will start a 2nd run which downloads them all.

If you have thousands of files, it may take a couple of minutes to list them all before it starts the download.

[nodecentral]

I do have a lot of files :-)

Does that check just look at the file names, or does it do anything more sophisticated?

The reason for asking is that I want to re-create my Docker container with more options, but still use the same storage locations again. If I did that will it avoid downloading all the photos again ?

QQ : As well as photos, can I download all the my documents/files & folders stored in Files ?

[boredazfcuk]

Does that check just look at the file names, or does it do anything more sophisticated?

Unfortunately not. It just logs into the icloud.com website, lists the names of the files, compares them against the files on disk, and downloads anything that it can't find.

The reason for asking is that I want to re-create my Docker container with more options, but still use the same storage locations again. If I did that will it avoid downloading all the photos again ?

As long as the new container has the same bind mount to your photo storage location, then it should be OK:

Volume from host    Mount Point      
/home/john/iCloud   /share/Backups/VM/icloudpd/photos   Read/Write

QQ : As well as photos, can I download all the my documents/files & folders stored in Files ?

Sorry, the underlying application is only capable of downloading photos. I don't have much to do with that, I just wrapped it up in a Docker container as I generally do that with all the applications I run.

[nodecentral]

As long as the new container has the same bind mount to your photo storage location, then it should be OK:

FYI - I've just created a new container with the same bind mounts, which picked up the ".mounted' file, and also the cookie i had created previous (great !) ... Unfortunately as I'd used a diffierent 'folder_structure' value (%Y) in my new docker configuration, it's sadly downloading every photo again.. Oh well.. :-(

For us Synology/QNAP users - when the 'Interactive_only" is set to 'True' , it would be nice to be able to have a seperate process, to create/recreate the Apple 2FA cookie and then have that written into the /config folder so it's ready/waiting to be used.

[boredazfcuk]

Unfortunately as I'd used a diffierent 'folder_structure' value (%Y) in my new docker configuration, it's sadly downloading every photo again.. Oh well.. :-(

Aw man... The amount of times I've done that isn't funny. I just clear out my whole folder and start again!

For us Synology/QNAP users - when the 'Interactive_only" is set to 'True' , it would be nice to be able to have a seperate process, to create/recreate the Apple 2FA cookie and then have that written into the /config folder so it's ready/waiting to be used.

It's a bit of a pain really. Back in the early days, the container always ran interactively. The problem with that was that you needed to spin up a second container to create the cookie. I wanted to get rid of that so that it could all be handled in a single process. To make it easier for people (running standard Docker containers that are non-interactive) I just made it so that when people are manually running the script, they get prompted for the 2FA info, if not, then it just synchronised photos. This worked pretty well until someone with a Synology device came along and I found that they don't work the same as the usual Docker install.

Anyway, I've added another command line option, --Generate2FACookie, which will just run the 2FA bit and then exit out. Details are in the Readme. I've pushed this new version to Dockerhub too so you'll need to update from there. Think this should make life a bit simpler for you further down the line.