search

steilerDev / icloud-photos-sync

One-way sync engine for the iCloud Photos Library into the native file system
https://icps.steiler.dev/
GNU General Public License v3.0
348 stars 9 forks source link

Advanced Data Protection Support #202

Open gsong opened 1 year ago

gsong commented 1 year ago

Describe the bug

$ docker compose exec photos-sync icloud-photos-sync token
-----------------------------------------------------------------------------------------------
Welcome to icloud-photos-sync, v.1.0.1!
Made with <3 by steilerDev
-----------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Sign in successful!
-----------------------------------------------------------------------------------------------
Experienced fatal error at 3/3/2023, 12:58:54 PM: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: 734410aa-a8db-48ce-971f-0f6041f8751a)
-----------------------------------------------------------------------------------------------

Note that I do have Advanced Data Protection turned on.

Logs Please paste the log file (preferably with LOG_LEVEL=debug), located in .icloud-photos-sync.log, stored in the DATA_DIR.

[2023-03-03T20:58:50.110Z] INFO i-Cloud: Initiating iCloud connection
[2023-03-03T20:58:50.652Z] INFO i-Cloud: Authenticating user
[2023-03-03T20:58:51.323Z] INFO i-Cloud: Authentication successful
[2023-03-03T20:58:51.328Z] INFO i-Cloud: Setting up iCloud connection
[2023-03-03T20:58:53.679Z] INFO i-Cloud: Getting iCloud Photos Service ready
[2023-03-03T20:58:54.627Z] ERROR Error-Handler: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: 734410aa-a8db-48ce-971f-0f6041f8751a)

Operating environmnent

steilerDev commented 1 year ago

@gsong are you using iCloud Shared Photo Library?

gsong commented 1 year ago

@gsong are you using iCloud Shared Photo Library?

No, I'm not.

krubenok commented 1 year ago

I'm in a similar position with advanced data protection and a yubikey on my Apple ID. With this combination of security settings, the sign in prompts I receive on other devices don't have MFA codes, but rather just "ok" or "that wasn't me".

Edit; this feels like something that might have a soft dependency on #120 to do yubikey auth via a webUI.

steilerDev commented 1 year ago

@gsong Any of those things enabled with you as well?

Otherwise your use case @krubenok would be a separate issue - however: I currently don't have an account setup like this (neither do I plan to do so). Most importantly, in a scenario, where Advanced Data Protection is enabled, the possibility to access your data from the WebUI needs to be enabled (this can be done optionally) - as this tool is using the APIs used by the WebUI.

In case you want me to investigate your use case further, I'd need an HAR file from your authentication against the iCloud WebUI - based on that I might be able to understand what needs to change in order to support this - #120 is not related. (Full disclosure: Keep in mind that this HAR file might contain sensitive data - unless you know how to purge it, you need to trust me that I won't abuse this - however since the MFA trust token is location/IP specific I probably won't be able to use the data from those requests anyway) See #207

steilerDev commented 1 year ago

I just realise I was skipping over this part @gsong

Note that I do have Advanced Data Protection turned on.

You will need to make sure that access through the iCloud WebUI is enabled. See Apple's support document on this. The tool is re-using those APIs (and I hope the APIs are the same when this is enabled, since I cannot test this).

Please report back - in case it does not work, I would need to ask the same of you as above.

steilerDev commented 1 year ago

@krubenok I've created #207 for the addition of YubiKey support - but I'll need help on that.

noah-guillory commented 1 year ago

Hiya, I'm trying to set this tool up and I have Advanced Data Protection enabled on my iCloud account. I'm currently getting the following error when I try running the token command:

APP_TOKEN (FATAL): Unable to acquire trust token caused by AUTH_FAILED (FATAL): iCloud Authentication failed caused by ICLOUD_PHOTOS_SETUP_FAILED (FATAL): Unable to get iCloud Photos service ready caused by ICLOUD_PHOTOS_SETUP_ERROR (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: cadf49b6-85c4-47b4-84a4-ced655765a67)

I've made sure that I have the "Access iCloud Data on the Web` option enabled in my account settings.

Let me know if I could provide you with more information to help troubleshoot this as I do not want to disable ADP.

steilerDev commented 1 year ago

@noah-guillory which 2FA method are you using.

Does the WebUI access work (have you ever accessed the UI from a non Safari browser, where you provided password instead of Touch ID)

noah-guillory commented 1 year ago

@noah-guillory which 2FA method are you using.

Does the WebUI access work (have you ever accessed the UI from a non Safari browser, where you provided password instead of Touch ID)

I am using the normal 2FA method, not using any hardware security keys or anything.

I was able to get through the process of providing my 2FA code by curling it to the MFA endpoint.

And I am able to access Photos from the WebUI using Edge as well. Though whenever I do, I do get a push notification on my Mac saying that my device is providing access to the iCloud web interface.

steilerDev commented 1 year ago

Do you need to confirm this notification before being able to continue?

steilerDev commented 1 year ago

I need to understand how the API behaves differently from the current process, when ADP is enabled.

Best way for me to debug is by being able to see the iCloud API's behaviour here. For that I'd need a HAR file of your login on the browser. For that do the following:

  1. Open a new private window in Chrome
  2. Navigate to icloud.com
  3. Open the Network Tab of the Developer Tools (e.g. right click on the page and select 'Inspect')
  4. On the Dev Tools Network select 'Preserve log' and 'Disable cache'
  5. Clear the log and make sure logs are recorded
  6. Perform login and open iCloud Photos
  7. Once done, 'Download har' file
  8. Sent it to me so I can take a look at what's happening - feel free to sent it to my email frank@steilerdev.de

Example of how to do this:

https://user-images.githubusercontent.com/7031616/230869102-3aef6cea-5554-422c-bca2-51bb7f81df6e.mp4

noah-guillory commented 1 year ago

Makes sense! Whenever I get a chance I'll get you that file. Thanks for being responsive 😄

Tomfox91 commented 1 year ago

Hi @steilerDev, is there any hope of ADP support landing soon? Did you get the input you needed?

steilerDev commented 1 year ago

@Tomfox91 unfortunately I have not received any feedback on my previous request - so I have not had the chance to implement this.

steilerDev commented 1 year ago

Thanks @Tomfox91 for sending over an HAR file - I just had a quick look - some things look different, but the good news is that the API is very close to what I am expecting :)

Unfortunately I'm not sure when I'll get around working on this as private and professional life are currently taking a lot of time :/ Anyone who wants to support on this, I'm happy to point you into directions :)

skaeight commented 1 year ago

Sadly I think the resolution to this issue is to buy a used / refurb m1 Mac mini.

steilerDev commented 1 year ago

@skaeight looking at the previously shared HAR files by @Tomfox91 I don't think this will be necessary (as long as you allow iCloud Access through the WebUI)

frprm commented 1 year ago

I tried a sync using 1.2.0-beta.4, it fails with this:

Error: APP_SYNC: Sync failed caused by AUTH_FAILED: iCloud Authentication failed caused by ICLOUD_PHOTOS_SETUP_FAILED: Unable to get iCloud Photos service ready caused by ICLOUD_PHOTOS_SETUP_ERROR: Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (error code: 67717c18-aaf8-4d72-9b1c-8b1a66068302)

I can also confirm that "Access iCloud Data on the Web" option is enabled in my account settings.

steilerDev commented 1 year ago

See https://github.com/foxt/icloud.js/issues/4 for some research done on this topic

github-actions[bot] commented 1 year ago

This issue should be resolved with version v1.3.0-beta.1, please confirm.

steilerDev commented 1 year ago

As you see, I've added the initial bits based on the available investigations - I'm pretty sure that it won't work straight away, but I can't test it myself.

So if someone with ADP enabled could run 1.3.0-beta.1 and report back (preferably with a network log), that would be great!

MaxNeedsSnacks commented 1 year ago

Getting a status 500 when the app is trying to request PCS cookies; looking at the network log reveals this response:

                    "content": {
                        "size": 68,
                        "mimeType": "application/json; charset=UTF-8",
                        "text": "{\"success\":false,\"error\":\"Invalid X-APPLE-WEBAUTH-HSA-LOGIN cookie\"}"
                    },

and indeed, looking at the request, I don't think you're setting that cookie correctly...

image

image

(above images redacted for obvious reasons)

steilerDev commented 1 year ago

I'm guessing the second screenshot is the reply from setup.icloud.com/setup/ws/1/accountLogin? Meaning we currently don't get a value for the HSA-LOGIN cookie?

MaxNeedsSnacks commented 1 year ago

Sorry, yep, I just noticed that as well, can't set what you don't get :p

Still weird that it gave me no HSA-LOGIN tho, this was on a fresh login if that helps (i.e. no cached trust token or anything)

steilerDev commented 1 year ago

Yeah - on a previously shared HAR file of the web interaction I can see this cookie getting set (and I don't get it for a non-ADP account). So this empty value feels like an application bug - but the HAR file should capture network traffic before processing, unless there is an issue with the underlying axios library parsing the request...

Have you retried to see if this is 'just' a flaky API?

MaxNeedsSnacks commented 1 year ago

Just tried and failed again, specifically tried both logging in again and using an established trust token :/

steilerDev commented 1 year ago

I've checked again - and I do get an WEBAUTH_HSA_LOGIN cookie on the web and none in the tool (but I don't seem to need it as long as I don't have ADP) - so I will need to dig into this...

steilerDev commented 1 year ago

Okay - so it seems the WEBAUTH_HSA_LOGIN token is only temporarily available and necessary to acquire the PCS cookies when ADP is enabled. I think this will require more re-implementation efforts than expected.

Before digging too deep into this: Given the fact, that it seems that ADP enabled accounts need to provide manual authorization to this tool at least every hour (this is how long the PCS cookies seem to be valid for), does it even make sense to support accounts like this with a continuous syncing tool?

I'm curious what the potential users (@MaxNeedsSnacks / @frprm / @noah-guillory / anyone else watching this issue) of this are saying? Especially on the initial run and/or on scheduled runs you'd need to confirm the web login every hour and at every sync - would you (under those constraints) even run this tool (I'm asking this as an honest question, because I wouldn't - and I don't want to waste time on a difficult feature that no one is going to use :D )

MaxNeedsSnacks commented 1 year ago

Hmm.. like I said before in another issue, it works for my usecase personally since the sync would just be occasional and all the authorisation is is a quick button push, but honestly, for continuous sync, I can't imagine many people using this unfortunately ^^;

steilerDev commented 1 year ago

363 is taking precedence for now - I'll be looking at ADP once I'm back following the current iCloud Web App authentication flow.

ido2 commented 1 year ago

With ADP enabled (and yes, I would like to run this manually once every, so this workflow of one time authentication suites me fine), I get the following error:

User authenticated Device trusted Advanced Data Protection requires additional cookies, acquiring... Error: APP_SYNC: Sync failed caused by AUTH_FAILED: iCloud Authentication failed caused by AUTH_PCS_REQUEST_FAILED: Unable to acquire PCS cookies caused by Request failed with status code 500 (error code: a08aa434-8ce7-4f80-b155-0d4a2914883f)

MaxNeedsSnacks commented 1 year ago

So side note from me a couple of months later, I found out rather quickly that the Windows iCloud client has no issues whatsoever with ADP when I switched back to it for work a while ago, so I do sort of wonder what Apple is doing differently over there. If it's something that would require reverse engineering the iCloud binaries, I... can't really recommend that because depending on jurisdiction and how litigious they're feeling that's a bit precarious, though I still wonder if there is a way to somehow acquire a more... permanent form of trust

steilerDev commented 1 year ago

That is a very interesting consideration @MaxNeedsSnacks - unfortunately I don't have much experience in reverse engineering windows binaries, but maybe we can capture network traffic through a proxy to understand what's going on.

I'm currently in the process of creating a new Apple ID that has ADP enabled so I can dig into this capability.

steilerDev commented 1 year ago

If anyone is having this setup, you could setup https://proxyman.io to capture the traffic and collaborate on what you are seeing!

steilerDev commented 1 year ago

Great news on this: After getting a fresh SIM and setting up a fresh account on an old iPhone that has ADP enabled, I've got a sync execution completed with this tool 🥳

This means it generally works and the changes I need to do to have this "production ready" are only minimal. Will have something shortly :)

steilerDev commented 1 year ago

Interestingly enough, the UI says it provides access "as long as the user is logged in". This is probably why the Windows Desktop client does not require extensive re-authentications.

I might see if there is an API I can hit continuously in the background to stay logged in longer than the current limit of 60 minutes, otherwise one would need to attend the sync and re-authorize every hour

MaxNeedsSnacks commented 1 year ago

That's probably not why Windows doesn't have to reauth though, since otherwise you'd have to log in again after having your PC shut down for more than an hour ^^;

steilerDev commented 1 year ago

Looking good - need to create tests and will then be able to release.

image

Currently the sync of large libraries might be a little bit painful (because I expect re-authorization every 60mins). For the release after next, I will focus on reworking the API interactions (see issue 364), which should also make using ADP more pleasant :)

steilerDev commented 12 months ago

1.4.0-nightly.1 should contain ADP support!

github-actions[bot] commented 12 months ago

This issue should be resolved with version v1.4.0-beta.1, please confirm.

ido2 commented 12 months ago

I tested nightly (v.1.4.0-nightly.4!), was able to authenticate, got half a dozen "Your Apple ID was used to sign in to iCloud via a web browser" emails from apple, but sync command fails:

[2023-11-27T17:50:01.010Z] DEBUG iCloud: Acquired signin secrets
[2023-11-27T17:50:01.010Z] DEBUG iCloud: Response status is 200, authentication successful - device trusted
[2023-11-27T17:50:01.010Z] DEBUG ResourceManager: Reading resource file from /opt/icloud-photos-library/.icloud-photos-sync
[2023-11-27T17:50:01.011Z] INFO iCloud: Setting up iCloud connection
[2023-11-27T17:50:01.901Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-HSA-LOGIN (domain icloud.com) with length 0
[2023-11-27T17:50:01.901Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-UNIQUE-CLIENT-ID (domain icloud.com) with length 6
[2023-11-27T17:50:01.901Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-LOGIN (domain icloud.com) with length 220
[2023-11-27T17:50:01.901Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-VALIDATE (domain icloud.com) with length 220
[2023-11-27T17:50:01.902Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-TOKEN (domain icloud.com) with length 220
[2023-11-27T17:50:01.902Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-USER (domain icloud.com) with length 23
[2023-11-27T17:50:01.902Z] DEBUG HeaderJar: Extracted cookie from response header: X_APPLE_WEB_KB-A9PSKFGM1GEJWLXQCBLTPZQTYLI (domain icloud.com) with length 220
[2023-11-27T17:50:01.902Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-DS-WEB-SESSION-TOKEN (domain icloud.com) with length 810
[2023-11-27T17:50:01.903Z] DEBUG NetworkManager: Setting photosUrl to https://p106-ckdatabasews.icloud.com:443
[2023-11-27T17:50:01.904Z] DEBUG iCloud: PCS required, acquiring...
[2023-11-27T17:50:01.904Z] INFO iCloud: Acquiring PCS cookies
[2023-11-27T17:50:02.633Z] DEBUG NetworkManager: Settling rate limiter queue...
[2023-11-27T17:50:02.633Z] DEBUG NetworkManager: Queue has settled!
[2023-11-27T17:50:02.633Z] DEBUG NetworkManager: Settling CCY limiter queue...
[2023-11-27T17:50:02.633Z] DEBUG NetworkManager: Queue has settled!
[2023-11-27T17:50:02.633Z] DEBUG EventManager: Removed 3 listeners for source iCloudPhotos
[2023-11-27T17:50:02.633Z] DEBUG EventManager: No more listeners for source iCloudPhotos registered
[2023-11-27T17:50:02.633Z] DEBUG EventManager: Removed 12 listeners for source iCloud
[2023-11-27T17:50:02.633Z] DEBUG EventManager: No more listeners for source iCloud registered
[2023-11-27T17:50:02.636Z] INFO ErrorHandler: Handling error APP_SYNC caused by AUTH_PCS_REQUEST_FAILED
[2023-11-27T17:50:02.636Z] ERROR RuntimeError: APP_SYNC: Sync failed caused by AUTH_FAILED: iCloud Authentication failed caused by AUTH_PCS_REQUEST_FAILED: Unable to acquire PCS cookies caused by Request failed with status code 500 (error code: Enable crash reporting for error code)
ido2 commented 12 months ago

Some additional information.

I waited some time and tried again. Now works fine.

Maybe since it tried a couple of times so fast and I hadn't had enough time to authorize on the phone, then it was banned for a while.

So maybe you ought to not try in a busy loop..

Welcome to icloud-photos-sync, v.1.4.0-nightly.4!
Made with <3 by steilerDev
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Starting sync at 11/27/2023, 8:57:51 PM
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
steilerDev commented 12 months ago

Yeah - it seems Apple is quite quickly at limiting authentication requests when ADP is enabled - I've ran into the 500 issue as well when testing.

I might adjust the retry timeout - that's good feedback

rggjan commented 7 months ago

Hmm... I just tried both the nightly and the beta, but I always get:

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Welcome to icloud-photos-sync, v.1.4.0-nightly.8!
Made with <3 by steilerDev
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
MFA code required
Listening for input on port 80

and then it stops. On my iPhone, I get a notification that my icloud account is being logged in, but without asking me to allow it. When accessing iCloud on the web, I get a similar notification, but with an "allow" button, and then it works fine.

steilerDev commented 7 months ago

Before ADP kicks in, you will need to provide your MFA code - this is a 6-digit code either pushed to your iDevice or phone number.

See in the docs on how to submit the code to the app: https://icps.steiler.dev/get-started/#submit-mfa-code

rggjan commented 7 months ago

Thanks for the quick answer. I never got a code like that on my iPhone or Mac. Maybe because i have two yubikeys as 2FA, which I usually need for login? (Although I have two trusted phone numbers defined as well...)

steilerDev commented 7 months ago

Ahh - Yubikey authentication is currently out of scope (since I don't have an account setup with this and I don't own one) - if you've got trusted phone numbers you can have the MFA code resent to them: https://icps.steiler.dev/get-started/#re-send-mfa-code - just provide method 'sms' and a phone number id (those start at 0 and go up - it depends on how many you've got, but they should provide an error if the id is invalid)

rggjan commented 7 months ago

Oh, thanks!

So I tried running "docker exec photos-sync resend_mfa sms 0", but I get:

[2024-04-26T11:50:33.596Z] INFO iCloud: Resending MFA code with 'SMS' (Number ID: 0)

[2024-04-26T11:50:33.596Z] DEBUG iCloud: Requesting MFA code via URL https://idmsa.apple.com/appleauth/auth/verify/phone with data {"phoneNumber":{"id":0},"mode":"sms"}

Resending MFA code via 'SMS' (Number ID: 0)...
[2024-04-26T11:50:34.233Z] DEBUG HeaderJar: Extracted scnt from response header with length 446

[2024-04-26T11:50:34.235Z] DEBUG HeaderJar: Extracted cookie from response header: dslang (domain apple.com) with length 5

[2024-04-26T11:50:34.235Z] DEBUG HeaderJar: Extracted cookie from response header: site (domain apple.com) with length 3

[2024-04-26T11:50:34.235Z] DEBUG HeaderJar: Extracted cookie from response header: dslang (domain apple.com) with length 5

[2024-04-26T11:50:34.235Z] DEBUG HeaderJar: Extracted cookie from response header: site (domain apple.com) with length 3

[2024-04-26T11:50:34.236Z] WARN RuntimeWarning: Error within MFA flow: MFA_RESEND_FAILED: Unable to resend MFA code caused by VALIDATOR_RESEND_MFA_PHONE_RESPONSE: Unable to parse and validate resend MFA phone response (must have required property 'trustedPhoneNumber' (/data))

Warning: MFA_RESEND_FAILED: Unable to resend MFA code caused by VALIDATOR_RESEND_MFA_PHONE_RESPONSE: Unable to parse and validate resend MFA phone response (must have required property 'trustedPhoneNumber' (/data))

Do you think it's possible that SMS 2FA is disabled by Apple when security keys are used?

steilerDev commented 7 months ago

Can you try with id 1/2/3...?

I know that deleting and re-adding a phone number will increase this id (my demo account always needs id '3' for some reason :D )

rggjan commented 7 months ago

I tried up to 6, no luck.

Looking at https://discussions.apple.com/thread/254617891?sortBy=best it seems weaker options (like SMS) are disabled when security keys are added.

Too bad... seems I have to make a choice between backups and strong security 😅