Can no longer log in: Bad username or password

boredazfcuk commented 9 months ago

Overview

As of about 45mins ago, all four of my containers attempted a download, and all four failed to login.

After removing the keyring file and attempting to recreate it with icloud --username my@email.com the password is rejected. I've confirmed the e-mail and password combination is correct.

Steps to Reproduce

Start container

Expected Behavior

Logs into icloud

Actual Behavior

Fails to login. If password is saved to the keyring, this error is generated:

  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 220, in authenticate
    req = self.session.post(
          ^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 105, in request
    self._raise_error(code, reason)
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 127, in _raise_error
    raise api_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/icloudpd_latest/bin/icloudpd", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/icloudpd/base.py", line 317, in main
    core(
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/icloudpd/base.py", line 744, in core
    icloud = authenticator(logger, domain)(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/icloudpd/authentication.py", line 31, in authenticate_
    icloud = pyicloud_ipd.PyiCloudService(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 204, in __init__
    self.authenticate()
  File "/opt/icloudpd_latest/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 228, in authenticate
    raise PyiCloudFailedLoginException(msg, error)
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason'))

Context

Seems that Apple may have changed something. Can replicate on Alpine Linux 3.18.3 icloudpd 1.16.2 and Alpine Linux 3.18.5 and icloudpd 1.16.3

sonicflame17 commented 9 months ago

I'm having the same issue. Hoping it's a temporary problem with iCloud itself. I was able to log in just yesterday with iCloudPD, and my configuration has not changed since then.

gunner007cd commented 9 months ago

Same issue as well, failure to login, even after changing password to something simple.

AndreyNikiforov commented 9 months ago

I can replicate the issue on docker with 1.16.3 without keyring, so it is general issue and most likely related to some changes on Apple side as my 1.16.3 was running for a number of days before the issue started.

OZidkani commented 9 months ago

Same here

dohgren78 commented 9 months ago

Same here

seankerrigan commented 9 months ago

also experiencing the same issue with 1.16.2 as well as after upgrading to 1.16.3

jckesser commented 9 months ago

Same here

benstn commented 9 months ago

Same issue - container has stopped downloading today

oliplot commented 9 months ago

Same issue here, stopped working yesterday Dec 6

jrodmonaco commented 9 months ago

Same. Following this thread for updates.

gxander85 commented 9 months ago

Following

petebocken commented 9 months ago

Same here, following

eastexxiao commented 9 months ago

Same, following;

staffankvisth commented 9 months ago

I also get the same error on two different iCloud accounts. Suddenly stopped working 2 days ago. Both of my boredazfcuk/docker-icloudpd containers stopped and also manual execution of the script ~/.local/bin/icloudpd -u my@email.address -p ** -d . --folder-structure={:%Y/%m/%d} --delete-after-download fails with: _2023-12-07 23:24:36 DEBUG Authenticating... 2023-12-07 23:24:36 ERROR Unknown reason Traceback (most recent call last): File "pyicloud_ipd/base.py", line 220, in authenticate File "requests/sessions.py", line 637, in post File "pyicloud_ipd/base.py", line 105, in request File "pyicloud_ipd/base.py", line 127, in _raise_error pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "starters/icloudpd.py", line 5, in File "click/core.py", line 1157, in call File "click/core.py", line 1078, in main File "click/core.py", line 1434, in invoke File "click/core.py", line 783, in invoke File "icloudpd/base.py", line 317, in main File "icloudpd/base.py", line 744, in core File "icloudpd/authentication.py", line 31, in authenticate_ File "pyicloud_ipd/base.py", line 204, in init File "pyicloud_ipd/base.py", line 228, in authenticate pyicloudipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason')) [1357605] Failed to execute script 'icloudpd' due to unhandled exception!

Logging in on the icloud.com website works as it should. Does this mean there is no remedy for this problem yet? Has Apple changed something?

syunlee66 commented 9 months ago

Same, following;

Designator-Ol commented 9 months ago

just following.

meilon commented 9 months ago

just following.

Then please just please subscribe, nobody wants to get spammed with "just following" updates!

MaPa80711 commented 9 months ago

Hi, maybe it will help to solve the problem. I have three icloud Accounts running and the issue is the same in every account.

icloudpd --directory /xxx/xxx/xxx/icloudpd_photo_backup --username xxxxxxxxxxxxx@icloud.com --password xxxxxxxx --log-level debug 2023-12-08 16:16:11 DEBUG Authenticating... 2023-12-08 16:16:12 ERROR Unknown reason Traceback (most recent call last): File "pyicloud_ipd/base.py", line 220, in authenticate File "requests/sessions.py", line 637, in post File "pyicloud_ipd/base.py", line 105, in request File "pyicloud_ipd/base.py", line 127, in _raise_error pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "starters/icloudpd.py", line 5, in File "click/core.py", line 1157, in call File "click/core.py", line 1078, in main File "click/core.py", line 1434, in invoke File "click/core.py", line 783, in invoke File "icloudpd/base.py", line 317, in main File "icloudpd/base.py", line 744, in core File "icloudpd/authentication.py", line 31, in authenticate_ File "pyicloud_ipd/base.py", line 204, in init File "pyicloud_ipd/base.py", line 228, in authenticate pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason')) [201833] Failed to execute script 'icloudpd' due to unhandled exception!

espro1 commented 9 months ago

I'm experiencing this too. Started on the 6th at around 5pm EST.

AndreyNikiforov commented 9 months ago

I see the following paths:

dig into current code/protocol to see if minor tweak will make it working
- rationale: fastest and least invasive
- concerns: low chances
- chances: low; suspect Apple turned off old protocol that icloudpd uses (they support OAUTH on web for some time)
- investments: low
use original pyicloud lib
- rationale: IIUC pyicloud was updated to support OAUTH some time ago
- concerns: may break some of our code
- chances: mid-high; need to confirm that pyicloud works with current Apple services first
- investments: mid; need to update tests and that has been a challenge in the prev attempts to bring latest pyicloud
use own code to support latest Apple auth
- rationale: guaranteed success, opens door for other long-requested improvements
- concerns: may force larger updates, leading to breaking changes, long alpha-beta cycle
- chances: high
- investments: high

Is anybody interested in 1) and/or 2)? I plan to allocate some time in the next number of days to look into this issue and will invest in 1 & 3. IMO it is okay for multiple ppl to look at the same path as there are always edge cases and different approaches that may give different or complementing outcomes.

cfurrow commented 9 months ago

@AndreyNikiforov Thanks for the summary / plans of attack.

To help guide others that may want to debug, a possible good test case to try is within the tests/test_authentication.py tests, test_2sa_required. Per the instructions in the code, one can delete the pre-recorded VCR response file tests/vcr_cassettes/auth_requires_2sa.yml, and fill in their actual username/password in the test, and then run scripts/test (SEE https://github.com/icloud-photos-downloader/icloud_photos_downloader/blob/master/CONTRIBUTING.md#setting-up-the-development-environment on how to setup your local environment to test this code)

The tests will fail, and the VCR recording (aka "cassette") will be updated with the actual response received from icloud.

BE SURE TO NOT SHARE YOUR USERNAME/PASSWORD WHICH ARE PRESENT IN THE VCR RECORDING!

For instance, after doing the above, this is the diff of the VCR cassette. I can't glean much useful information from the response yet:

My new yml file has lots of whitespace/formatting differences vs the original
The new response is less verbose than the expected "good" response
I'm not as familiar with Python or these icloud related Python libraries.

"git diff" of the VCR yaml file

```diff diff --git a/tests/vcr_cassettes/auth_requires_2sa.yml b/tests/vcr_cassettes/auth_requires_2sa.yml index d0bbf78..ad75ec9 100644 --- a/tests/vcr_cassettes/auth_requires_2sa.yml +++ b/tests/vcr_cassettes/auth_requires_2sa.yml @@ -1,37 +1,59 @@ interactions: - request: - body: !!python/unicode '{"apple_id": "jdoe@gmail.com", "password": "password1", - "extended_login": false}' + body: '{"apple_id": "jdoe@gmail.com", "password": "password1", "extended_login": + false}' headers: - Accept: ['*/*'] - Accept-Encoding: ['gzip, deflate'] - Connection: [keep-alive] - Content-Length: ['88'] - Origin: ['https://www.icloud.com'] - Referer: ['https://www.icloud.com/'] - User-Agent: [Opera/9.52 (X11; Linux i686; U; en)] + Accept: + - '*/*' + Accept-Encoding: + - gzip, deflate + Connection: + - keep-alive + Content-Length: + - '90' + Origin: + - https://www.icloud.com + Referer: + - https://www.icloud.com/ + User-Agent: + - Opera/9.52 (X11; Linux i686; U; en) method: POST - uri: https://setup.icloud.com/setup/ws/1/login?clientBuildNumber=17DHotfix5&clientMasteringNumber=17DHotfix5&clientId=EC5646DE-9423-11E8-BF21-14109FE0B321&ckjsVersion=2.0.5&ckjsBuildVersion=17DProjectDev77 + uri: https://setup.icloud.com/setup/ws/1/login?clientBuildNumber=17DHotfix5&clientMasteringNumber=17DHotfix5&ckjsBuildVersion=17DProjectDev77&ckjsVersion=2.0.5&clientId=EC5646DE-9423-11E8-BF21-14109FE0B321 response: - body: {string: !!python/unicode '{"dsInfo":{"lastName":"Doe","iCDPEnabled":false,"dsid":"123456789","hsaEnabled":true,"ironcadeMigrated":true,"locale":"en-us_US","brZoneConsolidated":false,"isManagedAppleID":false,"gilligan-invited":"true","appleIdAliases":["jdoe@icloud.com"],"hsaVersion":2,"isPaidDeveloper":true,"countryCode":"USA","notificationId":"12341234-1234-1234-1234-143241234123","primaryEmailVerified":true,"aDsID":"12341234123412341234","locked":false,"hasICloudQualifyingDevice":true,"primaryEmail":"jdoe@gmail.com","appleIdEntries":[{"isPrimary":true,"type":"EMAIL","value":"jdoe@gmail.com"}],"gilligan-enabled":"true","fullName":"John - Doe","languageCode":"en-us","appleId":"jdoe@gmail.com","firstName":"John","iCloudAppleIdAlias":"jdoe@icloud.com","notesMigrated":true,"hasPaymentInfo":true,"pcsDeleted":false,"appleIdAlias":"","brMigrated":true,"statusCode":2},"hasMinimumDeviceForPhotosWeb":true,"iCDPEnabled":false,"webservices":{"reminders":{"url":"https://p10-remindersws.icloud.com:443","status":"active"},"notes":{"url":"https://p10-notesws.icloud.com:443","status":"active"},"mail":{"url":"https://p10-mailws.icloud.com:443","status":"active"},"ckdatabasews":{"pcsRequired":true,"url":"https://p10-ckdatabasews.icloud.com:443","status":"active"},"photosupload":{"pcsRequired":true,"url":"https://p10-uploadphotosws.icloud.com:443","status":"active"},"photos":{"pcsRequired":true,"uploadUrl":"https://p10-uploadphotosws.icloud.com:443","url":"https://p10-photosws.icloud.com:443","status":"active"},"drivews":{"pcsRequired":true,"url":"https://p10-drivews.icloud.com:443","status":"active"},"uploadimagews":{"url":"https://p10-uploadimagews.icloud.com:443","status":"active"},"schoolwork":{},"cksharews":{"url":"https://p10-ckshare.icloud.com:443","status":"active"},"findme":{"url":"https://p10-fmipweb.icloud.com:443","status":"active"},"ckdeviceservice":{"url":"https://p10-ckdevice.icloud.com:443"},"iworkthumbnailws":{"url":"https://p10-iworkthumbnailws.icloud.com:443","status":"active"},"calendar":{"url":"https://p10-calendarws.icloud.com:443","status":"active"},"docws":{"pcsRequired":true,"url":"https://p10-docws.icloud.com:443","status":"active"},"settings":{"url":"https://p10-settingsws.icloud.com:443","status":"active"},"ubiquity":{"url":"https://p10-ubiquityws.icloud.com:443","status":"active"},"streams":{"url":"https://p10-streams.icloud.com:443","status":"active"},"keyvalue":{"url":"https://p10-keyvalueservice.icloud.com:443","status":"active"},"archivews":{"url":"https://p10-archivews.icloud.com:443","status":"active"},"push":{"url":"https://p10-pushws.icloud.com:443","status":"active"},"iwmb":{"url":"https://p10-iwmb.icloud.com:443","status":"active"},"iworkexportws":{"url":"https://p10-iworkexportws.icloud.com:443","status":"active"},"geows":{"url":"https://p10-geows.icloud.com:443","status":"active"},"account":{"iCloudEnv":{"shortId":"p","vipSuffix":"p"},"url":"https://p10-setup.icloud.com:443","status":"active"},"fmf":{"url":"https://p10-fmfweb.icloud.com:443","status":"active"},"contacts":{"url":"https://p10-contactsws.icloud.com:443","status":"active"}},"pcsEnabled":true,"configBag":{"urls":{"accountCreateUI":"https://appleid.apple.com/widget/account/?widgetKey=12312412412341234123412341234123412341234#!create","accountLoginUI":"https://idmsa.apple.com/appleauth/auth/signin?widgetKey=83545bf919730e51dbfba24e7e8a78d2","accountLogin":"https://setup.icloud.com/setup/ws/1/accountLogin","accountRepairUI":"https://appleid.apple.com/widget/account/?widgetKey=12312412412341234123412341234123412341234#!repair","downloadICloudTerms":"https://setup.icloud.com/setup/ws/1/downloadLiteTerms","repairDone":"https://setup.icloud.com/setup/ws/1/repairDone","vettingUrlForEmail":"https://id.apple.com/IDMSEmailVetting/vetShareEmail","accountCreate":"https://setup.icloud.com/setup/ws/1/createLiteAccount","getICloudTerms":"https://setup.icloud.com/setup/ws/1/getTerms","vettingUrlForPhone":"https://id.apple.com/IDMSEmailVetting/vetSharePhone"},"accountCreateEnabled":"true"},"hsaTrustedBrowser":false,"appsOrder":["mail","contacts","calendar","photos","iclouddrive","notes2","reminders","pages","numbers","keynote","newspublisher","fmf","find","settings"],"version":2,"isExtendedLogin":false,"pcsServiceIdentitiesIncluded":false,"hsaChallengeRequired":true,"requestInfo":{"country":"TH","timeZone":"GMT+7","isAppleInternal":true},"pcsDeleted":false,"iCloudInfo":{"SafariBookmarksHasMigratedToCloudKit":false},"apps":{"calendar":{},"reminders":{},"keynote":{"isQualifiedForBeta":true},"settings":{"canLaunchWithOneFactor":true},"mail":{},"numbers":{"isQualifiedForBeta":true},"photos":{},"pages":{"isQualifiedForBeta":true},"find":{"canLaunchWithOneFactor":true},"notes2":{},"iclouddrive":{},"newspublisher":{"isHidden":true},"fmf":{},"contacts":{}}}'} + body: + string: '{"success":false,"error":1}' headers: - access-control-allow-credentials: ['true'] - access-control-allow-origin: ['https://www.icloud.com'] - access-control-expose-headers: [X-Apple-Request-UUID, Via] - apple-originating-system: [UnknownOriginatingSystem] - apple-seq: ['0'] - apple-tk: ['false'] - cache-control: ['no-cache, no-store, private'] - connection: [keep-alive] - content-length: ['4895'] - content-type: [application/json; charset=UTF-8] - date: ['Mon, 30 Jul 2018 19:00:39 GMT'] - server: [AppleHttpServer/2f080fc0] - strict-transport-security: [max-age=31536000; includeSubDomains] - via: ['icloudedge:si03p00ic-ztde010417:7401:18RC341:Singapore'] - x-apple-jingle-correlation-key: [SJHIUN7879234KJHH8JBH] - x-apple-request-uuid: [NISUHFIOSUHFOSIDUHFOSIDF] - x-responding-instance: ['setupservice:328457238759328579234875'] - status: {code: 200, message: OK} + Access-Control-Allow-Credentials: + - 'true' + Access-Control-Allow-Origin: + - https://www.icloud.com + Cache-Control: + - no-cache, no-store, private + Connection: + - keep-alive + Content-Type: + - application/json; charset=UTF-8 + Date: + - Fri, 08 Dec 2023 16:37:26 GMT + Server: + - AppleHttpServer/78689afb4479 + Strict-Transport-Security: + - max-age=31536000; includeSubDomains + X-Apple-Edge-Response-Time: + - '45' + X-Apple-Request-UUID: + - e7626cfc-bb0c-4ccd-986d-b9327f8bcef3 + X-Responding-Instance: + - setupservice:44700503:pv51p47ic-qukt21032501:8003:2404B363:5934c9004de5 + access-control-expose-headers: + - X-Apple-Request-UUID,Via + content-length: + - '27' + via: + - 631194250daa17e24277dea86cf30319:7ecc2ff259156058e890ba5cf00be461:uschi7 + x-apple-user-partition: + - '47' + status: + code: 421 + message: Misdirected Request version: 1 ```

However, the HTTP response code of "421: Misdirected Request" may be the key, and indicate that icloud has updated its API and therefore pyicloud needs updating to reflect that.

Back in Oct 2023, someone submitted an issue to pyicloud regarding the 421 response: https://github.com/picklepete/pyicloud/issues/441, and 4 days ago there is a possible fix: https://github.com/picklepete/pyicloud/issues/406 (it's not merged yet)

My system:

macOS Sonoma 14.1.2 (Apple M3 Macbook Pro)
Python 3.12.0

robflate commented 9 months ago

Total speculation but I wonder if this has anything to do with Apple's attempts to block iMessage on Android;

https://www.theverge.com/2023/12/8/23994089/apple-beeper-mini-android-blocked-imessage-app

nscheer commented 9 months ago

My guess is that it has something to do with the recent support for passkey logins. At least for the iCloud website it asks for passkey and logging in via password takes one more click.

Such a thing would be unusual for an api, nevertheless a change that requires specification of the desired login method sounds very likely.

cutzenfriend commented 9 months ago

My guess is that it has something to do with the recent support for passkey logins. At least for the iCloud website it asks for passkey and logging in via password takes one more click.

Such a thing would be unusual for an api, nevertheless a change that requires specification of the desired login method sounds very likely.

This sounds like the issue for me...

charlesb87 commented 9 months ago

Same error here running on Windows x64 with 1.16.3 version

dlang123 commented 9 months ago

Same issue for me as well, docker that was previously working fine has now failed authorization today as well, and I haven't been able to authorize one for 3 days.

mattryles commented 9 months ago

This looks related to beeper mini being blocked. 😔

scaraebeus commented 9 months ago

I see the following paths:

dig into current code/protocol to see if minor tweak will make it working

rationale: fastest and least invasive

concerns: low chances

chances: low; suspect Apple turned off old protocol that icloudpd uses (they support OAUTH on web for some time)

investments: low

use original pyicloud lib

rationale: IIUC pyicloud was updated to support OAUTH some time ago

concerns: may break some of our code

chances: mid-high; need to confirm that pyicloud works with current Apple services first

investments: mid; need to update tests and that has been a challenge in the prev attempts to bring latest pyicloud

[ . . . ]

Is anybody interested in 1) and/or 2)? I plan to allocate some time in the next number of days to look into this issue and will invest in 1 & 3. IMO it is okay for multiple ppl to look at the same path as there are always edge cases and different approaches that may give different or complementing outcomes.

If it helps, I was able to verify the current 1.0.0 version of pyicloud works as is. I was able to instantiate a PyiCloudService and list albums (pyicloud,photos.albums).

Potentially doing a combination of path 1) and 2) may be an option - possibly just focusing on merging over the key authentication pieces from the 1.0.0 pyicloud.

cfurrow commented 9 months ago

If it helps, I was able to verify the current 1.0.0 version of pyicloud works as is. I was able to instantiate a PyiCloudService and list albums (pyicloud,photos.albums).

I was noticing that as well. icloud_photos_downloader uses some customized versions of pyicloud code to be compatible with China vs US icloud domains. It's possible those customizations are now causing some issues, and may not be updated with the latest code from pyicloud.

I think changes would have to be made to the following files:

src/pyicloud_ipd/base.py
- Needs to be updated to reflect latest version from pyicloud repo
- Note: this update will remove the domain-specific code changes made by icloud_photos_downloader to switch between US and China icloud domain names. That will have to be addressed.
src/icloudpd/authentication.py
- Need to be updated to call pyicloud_ipd.PyiCloudService(...) with correct arguments, removing "domain" from the arg list, etc.
src/pyicloud_ipd/exceptions.py
- Needs to be updated to reflect exception classes from pyicloud repo

"git diff" example that gets icloudpd_ipd updated with latest changes from PyiCloud

```diff diff --git a/src/pyicloud_ipd/base.py b/src/pyicloud_ipd/base.py index a387a65..e53f6f5 100644 --- a/src/pyicloud_ipd/base.py +++ b/src/pyicloud_ipd/base.py @@ -1,15 +1,13 @@ -import six -import uuid -import hashlib +from uuid import uuid1 import inspect import json import logging -import requests -import sys -import tempfile -import os +from requests import Session +from tempfile import gettempdir +from os import path, mkdir from re import match -import urllib3 +import http.cookiejar as cookielib +import getpass from pyicloud_ipd.exceptions import ( PyiCloudConnectionException, @@ -29,13 +27,15 @@ from pyicloud_ipd.services import ( ) from pyicloud_ipd.utils import get_password_from_keyring -if six.PY3: - import http.cookiejar as cookielib -else: - import cookielib +LOGGER = logging.getLogger(__name__) - -logger = logging.getLogger(__name__) +HEADER_DATA = { + "X-Apple-ID-Account-Country": "account_country", + "X-Apple-ID-Session-Id": "session_id", + "X-Apple-Session-Token": "session_token", + "X-Apple-TwoSV-Trust-Token": "trust_token", + "scnt": "scnt", +} class PyiCloudPasswordFilter(logging.Filter): @@ -51,329 +51,554 @@ class PyiCloudPasswordFilter(logging.Filter): return True -class PyiCloudSession(requests.Session): +class PyiCloudSession(Session): + """iCloud session.""" + def __init__(self, service): self.service = service - super(PyiCloudSession, self).__init__() + super().__init__() - def request(self, *args, **kwargs): + def request(self, method, url, **kwargs): # pylint: disable=arguments-differ # Charge logging to the right service endpoint callee = inspect.stack()[2] module = inspect.getmodule(callee[0]) - logger = logging.getLogger(module.__name__).getChild('http') - if self.service._password_filter not in logger.filters: - logger.addFilter(self.service._password_filter) - - logger.debug("%s %s %s", args[0], args[1], kwargs.get('data', '')) - - try: - response = super(PyiCloudSession, self).request(*args, **kwargs) - except requests.exceptions.SSLError: - raise PyiCloudConnectionException("Error establishing secure connection. Try --domain parameter") - - content_type = response.headers.get('Content-Type', '').split(';')[0] - json_mimetypes = ['application/json', 'text/json'] + request_logger = logging.getLogger(module.__name__).getChild("http") + if self.service.password_filter not in request_logger.filters: + request_logger.addFilter(self.service.password_filter) + + request_logger.debug("%s %s %s", method, url, kwargs.get("data", "")) + + has_retried = kwargs.get("retried") + kwargs.pop("retried", None) + response = super().request(method, url, **kwargs) + + content_type = response.headers.get("Content-Type", "").split(";")[0] + json_mimetypes = ["application/json", "text/json"] + + for header, value in HEADER_DATA.items(): + if response.headers.get(header): + session_arg = value + self.service.session_data.update( + {session_arg: response.headers.get(header)} + ) + + # Save session_data to file + with open(self.service.session_path, "w", encoding="utf-8") as outfile: + json.dump(self.service.session_data, outfile) + LOGGER.debug("Saved session data to file") + + # Save cookies to file + self.cookies.save(ignore_discard=True, ignore_expires=True) + LOGGER.debug("Cookies saved to %s", self.service.cookiejar_path) + + if not response.ok and ( + content_type not in json_mimetypes + or response.status_code in [421, 450, 500] + ): + try: + # pylint: disable=protected-access + fmip_url = self.service._get_webservice_url("findme") + if ( + has_retried is None + and response.status_code in [421, 450, 500] + and fmip_url in url + ): + # Handle re-authentication for Find My iPhone + LOGGER.debug("Re-authenticating Find My iPhone service") + try: + # If 450, authentication requires a full sign in to the account + service = None if response.status_code == 450 else "find" + self.service.authenticate(True, service) + + except PyiCloudAPIResponseException: + LOGGER.debug("Re-authentication failed") + kwargs["retried"] = True + return self.request(method, url, **kwargs) + except Exception: + pass + + if has_retried is None and response.status_code in [421, 450, 500]: + api_error = PyiCloudAPIResponseException( + response.reason, response.status_code, retry=True + ) + request_logger.debug(api_error) + kwargs["retried"] = True + return self.request(method, url, **kwargs) - if not response.ok and content_type not in json_mimetypes: self._raise_error(response.status_code, response.reason) if content_type not in json_mimetypes: return response try: - json = response.json() - except: - logger.warning('Failed to parse response with JSON mimetype') + data = response.json() + except: # pylint: disable=bare-except + request_logger.warning("Failed to parse response with JSON mimetype") return response - logger.debug(json) + request_logger.debug(data) - reason = json.get('errorMessage') - reason = reason or json.get('reason') - reason = reason or json.get('errorReason') - if not reason and isinstance(json.get('error'), six.string_types): - reason = json.get('error') - if not reason and json.get('error'): - reason = "Unknown reason" + if isinstance(data, dict): + reason = data.get("errorMessage") + reason = reason or data.get("reason") + reason = reason or data.get("errorReason") + if not reason and isinstance(data.get("error"), str): + reason = data.get("error") + if not reason and data.get("error"): + reason = "Unknown reason" - code = json.get('errorCode') - if not code and json.get('serverErrorCode'): - code = json.get('serverErrorCode') + code = data.get("errorCode") + if not code and data.get("serverErrorCode"): + code = data.get("serverErrorCode") - if reason: - self._raise_error(code, reason) + if reason: + self._raise_error(code, reason) return response def _raise_error(self, code, reason): - if self.service.requires_2sa and \ - reason == 'Missing X-APPLE-WEBAUTH-TOKEN cookie': - raise PyiCloud2SARequiredError(response.url) - if code == 'ZONE_NOT_FOUND' or code == 'AUTHENTICATION_FAILED': - reason = 'Please log into https://icloud.com/ to manually ' \ - 'finish setting up your iCloud service' - api_error = PyiCloudServiceNotActivatedErrror(reason, code) - logger.error(api_error) - - raise(api_error) - if code == 'ACCESS_DENIED': - reason = reason + '. Please wait a few minutes then try ' \ - 'again. The remote servers might be trying to ' \ - 'throttle requests.' - - api_error = PyiCloudAPIResponseError(reason, code) - logger.error(api_error) + if ( + self.service.requires_2sa + and reason == "Missing X-APPLE-WEBAUTH-TOKEN cookie" + ): + raise PyiCloud2SARequiredException(self.service.user["apple_id"]) + if code in ("ZONE_NOT_FOUND", "AUTHENTICATION_FAILED"): + reason = ( + "Please log into https://icloud.com/ to manually " + "finish setting up your iCloud service" + ) + api_error = PyiCloudServiceNotActivatedException(reason, code) + LOGGER.error(api_error) + + raise (api_error) + if code == "ACCESS_DENIED": + reason = ( + reason + ". Please wait a few minutes then try again." + "The remote servers might be trying to throttle requests." + ) + if code in [421, 450, 500]: + reason = "Authentication required for Account." + + api_error = PyiCloudAPIResponseException(reason, code) + LOGGER.error(api_error) raise api_error -class PyiCloudService(object): +class PyiCloudService: """ A base authentication class for the iCloud service. Handles the authentication required to access iCloud services. Usage: - from pyicloud_ipd import PyiCloudService + from pyicloud import PyiCloudService pyicloud = PyiCloudService('username@apple.com', 'password') - pyicloud_ipd.iphone.location() + pyicloud.iphone.location() """ + AUTH_ENDPOINT = "https://idmsa.apple.com/appleauth/auth" + HOME_ENDPOINT = "https://www.icloud.com" + SETUP_ENDPOINT = "https://setup.icloud.com/setup/ws/1" + def __init__( - self, domain, apple_id, password=None, cookie_directory=None, verify=True, - client_id=None + self, + apple_id, + password=None, + cookie_directory=None, + verify=True, + client_id=None, + with_family=True, ): if password is None: password = get_password_from_keyring(apple_id) + self.user = {"accountName": apple_id, "password": password} self.data = {} - self.client_id = client_id or str(uuid.uuid1()).upper() - self.user = {'apple_id': apple_id, 'password': password} - - self._password_filter = PyiCloudPasswordFilter(password) - logger.addFilter(self._password_filter) - - if (domain == 'com'): - self._home_endpoint = 'https://www.icloud.com' - self._setup_endpoint = 'https://setup.icloud.com/setup/ws/1' - elif (domain == 'cn'): - self._home_endpoint = 'https://www.icloud.com.cn' - self._setup_endpoint = 'https://setup.icloud.com.cn/setup/ws/1' - else: - raise NotImplementedError(f"Domain '{domain}' is not supported yet") + self.params = {} + self.client_id = client_id or ("auth-%s" % str(uuid1()).lower()) + self.with_family = with_family - self._base_login_url = '%s/login' % self._setup_endpoint + self.password_filter = PyiCloudPasswordFilter(password) + LOGGER.addFilter(self.password_filter) if cookie_directory: - self._cookie_directory = os.path.expanduser( - os.path.normpath(cookie_directory) - ) + self._cookie_directory = path.expanduser(path.normpath(cookie_directory)) + if not path.exists(self._cookie_directory): + mkdir(self._cookie_directory, 0o700) else: - self._cookie_directory = os.path.join( - tempfile.gettempdir(), - 'pyicloud', - ) + topdir = path.join(gettempdir(), "pyicloud") + self._cookie_directory = path.join(topdir, getpass.getuser()) + if not path.exists(topdir): + mkdir(topdir, 0o777) + if not path.exists(self._cookie_directory): + mkdir(self._cookie_directory, 0o700) + + LOGGER.debug("Using session file %s", self.session_path) + + self.session_data = {} + try: + with open(self.session_path, encoding="utf-8") as session_f: + self.session_data = json.load(session_f) + except: # pylint: disable=bare-except + LOGGER.info("Session file does not exist") + if self.session_data.get("client_id"): + self.client_id = self.session_data.get("client_id") + else: + self.session_data.update({"client_id": self.client_id}) self.session = PyiCloudSession(self) self.session.verify = verify - self.session.headers.update({ - 'Origin': self._home_endpoint, - 'Referer': '%s/' % self._home_endpoint, - 'User-Agent': 'Opera/9.52 (X11; Linux i686; U; en)' - }) + self.session.headers.update( + {"Origin": self.HOME_ENDPOINT, "Referer": "%s/" % self.HOME_ENDPOINT} + ) - cookiejar_path = self._get_cookiejar_path() + cookiejar_path = self.cookiejar_path self.session.cookies = cookielib.LWPCookieJar(filename=cookiejar_path) - if os.path.exists(cookiejar_path): + if path.exists(cookiejar_path): try: - self.session.cookies.load() - logger.debug("Read cookies from %s", cookiejar_path) - except: + self.session.cookies.load(ignore_discard=True, ignore_expires=True) + LOGGER.debug("Read cookies from %s", cookiejar_path) + except: # pylint: disable=bare-except # Most likely a pickled cookiejar from earlier versions. # The cookiejar will get replaced with a valid one after # successful authentication. - logger.warning("Failed to read cookiejar %s", cookiejar_path) - - self.params = { - 'clientBuildNumber': '17DHotfix5', - 'clientMasteringNumber': '17DHotfix5', - 'ckjsBuildVersion': '17DProjectDev77', - 'ckjsVersion': '2.0.5', - 'clientId': self.client_id, - } + LOGGER.warning("Failed to read cookiejar %s", cookiejar_path) self.authenticate() - def authenticate(self): + self._drive = None + self._files = None + self._photos = None + + def authenticate(self, force_refresh=False, service=None): """ - Handles authentication, and persists the X-APPLE-WEB-KB cookie so that + Handles authentication, and persists cookies so that subsequent logins will not cause additional e-mails from Apple. """ - logger.info("Authenticating as %s", self.user['apple_id']) - - data = dict(self.user) + login_successful = False + if self.session_data.get("session_token") and not force_refresh: + LOGGER.debug("Checking session token validity") + try: + self.data = self._validate_token() + login_successful = True + except PyiCloudAPIResponseException: + LOGGER.debug("Invalid authentication token, will log in from scratch.") + + if not login_successful and service is not None: + app = self.data["apps"][service] + if "canLaunchWithOneFactor" in app and app["canLaunchWithOneFactor"]: + LOGGER.debug( + "Authenticating as %s for %s", self.user["accountName"], service + ) + try: + self._authenticate_with_credentials_service(service) + login_successful = True + except Exception: + LOGGER.debug( + "Could not log into service. Attempting brand new login." + ) + + if not login_successful: + LOGGER.debug("Authenticating as %s", self.user["accountName"]) + + data = dict(self.user) + + data["rememberMe"] = True + data["trustTokens"] = [] + if self.session_data.get("trust_token"): + data["trustTokens"] = [self.session_data.get("trust_token")] + + headers = self._get_auth_headers() + + if self.session_data.get("scnt"): + headers["scnt"] = self.session_data.get("scnt") + + if self.session_data.get("session_id"): + headers["X-Apple-ID-Session-Id"] = self.session_data.get("session_id") - # We authenticate every time, so "remember me" is not needed - data.update({'extended_login': False}) + try: + self.session.post( + "%s/signin" % self.AUTH_ENDPOINT, + params={"isRememberMeEnabled": "true"}, + data=json.dumps(data), + headers=headers, + ) + except PyiCloudAPIResponseException as error: + msg = "Invalid email/password combination." + raise PyiCloudFailedLoginException(msg, error) from error + + self._authenticate_with_token() + + self._webservices = self.data["webservices"] + + LOGGER.debug("Authentication completed successfully") + + def _authenticate_with_token(self): + """Authenticate using session token.""" + data = { + "accountCountryCode": self.session_data.get("account_country"), + "dsWebAuthToken": self.session_data.get("session_token"), + "extended_login": True, + "trustToken": self.session_data.get("trust_token", ""), + } try: req = self.session.post( - self._base_login_url, - params=self.params, - data=json.dumps(data) + "%s/accountLogin" % self.SETUP_ENDPOINT, data=json.dumps(data) ) - resp = req.json() - except PyiCloudAPIResponseError as error: - msg = 'Invalid email/password combination.' - raise PyiCloudFailedLoginException(msg, error) - - # {'domainToUse': 'iCloud.com'} - domain_to_use = resp.get('domainToUse') - if domain_to_use != None: - msg = f'Apple insists on using {domain_to_use} for your request. Please use --domain parameter' - raise PyiCloudConnectionException(msg) + self.data = req.json() + except PyiCloudAPIResponseException as error: + msg = "Invalid authentication token." + raise PyiCloudFailedLoginException(msg, error) from error + + def _authenticate_with_credentials_service(self, service): + """Authenticate to a specific service using credentials.""" + data = { + "appName": service, + "apple_id": self.user["accountName"], + "password": self.user["password"], + } - self.params.update({'dsid': resp['dsInfo']['dsid']}) + try: + self.session.post( + "%s/accountLogin" % self.SETUP_ENDPOINT, data=json.dumps(data) + ) - if not os.path.exists(self._cookie_directory): - os.mkdir(self._cookie_directory) - self.session.cookies.save() - logger.debug("Cookies saved to %s", self._get_cookiejar_path()) + self.data = self._validate_token() + except PyiCloudAPIResponseException as error: + msg = "Invalid email/password combination." + raise PyiCloudFailedLoginException(msg, error) from error - self.data = resp - self.webservices = self.data['webservices'] + def _validate_token(self): + """Checks if the current access token is still valid.""" + LOGGER.debug("Checking session token validity") + try: + req = self.session.post("%s/validate" % self.SETUP_ENDPOINT, data="null") + LOGGER.debug("Session token is still valid") + return req.json() + except PyiCloudAPIResponseException as err: + LOGGER.debug("Invalid authentication token") + raise err + + def _get_auth_headers(self, overrides=None): + headers = { + "Accept": "*/*", + "Content-Type": "application/json", + "X-Apple-OAuth-Client-Id": "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d", + "X-Apple-OAuth-Client-Type": "firstPartyAuth", + "X-Apple-OAuth-Redirect-URI": "https://www.icloud.com", + "X-Apple-OAuth-Require-Grant-Code": "true", + "X-Apple-OAuth-Response-Mode": "web_message", + "X-Apple-OAuth-Response-Type": "code", + "X-Apple-OAuth-State": self.client_id, + "X-Apple-Widget-Key": "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d", + } + if overrides: + headers.update(overrides) + return headers - logger.info("Authentication completed successfully") - logger.debug(self.params) + @property + def cookiejar_path(self): + """Get path for cookiejar file.""" + return path.join( + self._cookie_directory, + "".join([c for c in self.user.get("accountName") if match(r"\w", c)]), + ) - def _get_cookiejar_path(self): - # Get path for cookiejar file - return os.path.join( + @property + def session_path(self): + """Get path for session data file.""" + return path.join( self._cookie_directory, - ''.join([c for c in self.user.get('apple_id') if match(r'\w', c)]) + "".join([c for c in self.user.get("accountName") if match(r"\w", c)]) + + ".session", ) @property def requires_2sa(self): - """ Returns True if two-step authentication is required.""" - return self.data.get('hsaChallengeRequired', False) \ - and self.data['dsInfo'].get('hsaVersion', 0) >= 1 - # FIXME: Implement 2FA for hsaVersion == 2 + """Returns True if two-step authentication is required.""" + return self.data.get("dsInfo", {}).get("hsaVersion", 0) >= 1 and ( + self.data.get("hsaChallengeRequired", False) or not self.is_trusted_session + ) + + @property + def requires_2fa(self): + """Returns True if two-factor authentication is required.""" + return self.data["dsInfo"].get("hsaVersion", 0) == 2 and ( + self.data.get("hsaChallengeRequired", False) or not self.is_trusted_session + ) + + @property + def is_trusted_session(self): + """Returns True if the session is trusted.""" + return self.data.get("hsaTrustedBrowser", False) @property def trusted_devices(self): - """ Returns devices trusted for two-step authentication.""" + """Returns devices trusted for two-step authentication.""" request = self.session.get( - '%s/listDevices' % self._setup_endpoint, - params=self.params + "%s/listDevices" % self.SETUP_ENDPOINT, params=self.params ) - return request.json().get('devices') + return request.json().get("devices") def send_verification_code(self, device): - """ Requests that a verification code is sent to the given device""" + """Requests that a verification code is sent to the given device.""" data = json.dumps(device) request = self.session.post( - '%s/sendVerificationCode' % self._setup_endpoint, + "%s/sendVerificationCode" % self.SETUP_ENDPOINT, params=self.params, - data=data + data=data, ) - return request.json().get('success', False) + return request.json().get("success", False) def validate_verification_code(self, device, code): - """ Verifies a verification code received on a trusted device""" - device.update({ - 'verificationCode': code, - 'trustBrowser': True - }) + """Verifies a verification code received on a trusted device.""" + device.update({"verificationCode": code, "trustBrowser": True}) data = json.dumps(device) try: - request = self.session.post( - '%s/validateVerificationCode' % self._setup_endpoint, + self.session.post( + "%s/validateVerificationCode" % self.SETUP_ENDPOINT, params=self.params, - data=data + data=data, ) - except PyiCloudAPIResponseError as error: + except PyiCloudAPIResponseException as error: if error.code == -21669: # Wrong verification code return False raise - # Re-authenticate, which will both update the HSA data, and - # ensure that we save the X-APPLE-WEBAUTH-HSA-TRUST cookie. - self.authenticate() + self.trust_session() return not self.requires_2sa + def validate_2fa_code(self, code): + """Verifies a verification code received via Apple's 2FA system (HSA2).""" + data = {"securityCode": {"code": code}} + + headers = self._get_auth_headers({"Accept": "application/json"}) + + if self.session_data.get("scnt"): + headers["scnt"] = self.session_data.get("scnt") + + if self.session_data.get("session_id"): + headers["X-Apple-ID-Session-Id"] = self.session_data.get("session_id") + + try: + self.session.post( + "%s/verify/trusteddevice/securitycode" % self.AUTH_ENDPOINT, + data=json.dumps(data), + headers=headers, + ) + except PyiCloudAPIResponseException as error: + if error.code == -21669: + # Wrong verification code + LOGGER.error("Code verification failed.") + return False + raise + + LOGGER.debug("Code verification successful.") + + self.trust_session() + return not self.requires_2sa + + def trust_session(self): + """Request session trust to avoid user log in going forward.""" + headers = self._get_auth_headers() + + if self.session_data.get("scnt"): + headers["scnt"] = self.session_data.get("scnt") + + if self.session_data.get("session_id"): + headers["X-Apple-ID-Session-Id"] = self.session_data.get("session_id") + + try: + self.session.get( + f"{self.AUTH_ENDPOINT}/2sv/trust", + headers=headers, + ) + self._authenticate_with_token() + return True + except PyiCloudAPIResponseException: + LOGGER.error("Session trust failed.") + return False + + def _get_webservice_url(self, ws_key): + """Get webservice URL, raise an exception if not exists.""" + if self._webservices.get(ws_key) is None: + raise PyiCloudServiceNotActivatedException( + "Webservice not available", ws_key + ) + return self._webservices[ws_key]["url"] + @property def devices(self): - """ Return all devices.""" - service_root = self.webservices['findme']['url'] + """Returns all devices.""" + service_root = self._get_webservice_url("findme") return FindMyiPhoneServiceManager( - service_root, - self.session, - self.params - ) - - @property - def account(self): - service_root = self.webservices['account']['url'] - return AccountService( - service_root, - self.session, - self.params + service_root, self.session, self.params, self.with_family ) @property def iphone(self): + """Returns the iPhone.""" return self.devices[0] + @property + def account(self): + """Gets the 'Account' service.""" + service_root = self._get_webservice_url("account") + return AccountService(service_root, self.session, self.params) + @property def files(self): - if not hasattr(self, '_files'): - service_root = self.webservices['ubiquity']['url'] - self._files = UbiquityService( - service_root, - self.session, - self.params - ) + """Gets the 'File' service.""" + if not self._files: + service_root = self._get_webservice_url("ubiquity") + self._files = UbiquityService(service_root, self.session, self.params) return self._files @property def photos(self): - if not hasattr(self, '_photos'): - service_root = self.webservices['ckdatabasews']['url'] - self._photos = PhotosService( - service_root, - self.session, - self.params - ) + """Gets the 'Photo' service.""" + if not self._photos: + service_root = self._get_webservice_url("ckdatabasews") + self._photos = PhotosService(service_root, self.session, self.params) return self._photos @property def calendar(self): - service_root = self.webservices['calendar']['url'] + """Gets the 'Calendar' service.""" + service_root = self._get_webservice_url("calendar") return CalendarService(service_root, self.session, self.params) @property def contacts(self): - service_root = self.webservices['contacts']['url'] + """Gets the 'Contacts' service.""" + service_root = self._get_webservice_url("contacts") return ContactsService(service_root, self.session, self.params) @property def reminders(self): - service_root = self.webservices['reminders']['url'] + """Gets the 'Reminders' service.""" + service_root = self._get_webservice_url("reminders") return RemindersService(service_root, self.session, self.params) - def __unicode__(self): - return 'iCloud API: %s' % self.user.get('apple_id') + @property + def drive(self): + """Gets the 'Drive' service.""" + if not self._drive: + self._drive = DriveService( + service_root=self._get_webservice_url("drivews"), + document_root=self._get_webservice_url("docws"), + session=self.session, + params=self.params, + ) + return self._drive def __str__(self): - as_unicode = self.__unicode__() - if sys.version_info[0] >= 3: - return as_unicode - else: - return as_unicode.encode('ascii', 'ignore') + return f"iCloud API: {self.user.get('apple_id')}" def __repr__(self): - return '<%s>' % str(self) + return f"<{self}>" diff --git a/src/icloudpd/authentication.py b/src/icloudpd/authentication.py index e92ea43..861b03a 100644 --- a/src/icloudpd/authentication.py +++ b/src/icloudpd/authentication.py @@ -29,8 +29,7 @@ def authenticator(logger: logging.Logger, domain: str): # If password not provided on command line variable will be set to None # and PyiCloud will attempt to retrieve from its keyring icloud = pyicloud_ipd.PyiCloudService( - domain, - username, password, + username, password=password, cookie_directory=cookie_directory, client_id=client_id, ) diff --git a/src/pyicloud_ipd/exceptions.py b/src/pyicloud_ipd/exceptions.py index ef9c917..80b7a69 100644 --- a/src/pyicloud_ipd/exceptions.py +++ b/src/pyicloud_ipd/exceptions.py @@ -1,39 +1,50 @@ - -class PyiCloudException(Exception): - pass +"""Library exceptions.""" -class PyiCloudConnectionException(PyiCloudException): - pass - -class PyiCloudNoDevicesException(PyiCloudException): +class PyiCloudException(Exception): + """Generic iCloud exception.""" pass -class PyiCloudAPIResponseError(PyiCloudException): - def __init__(self, reason, code): +# API +class PyiCloudAPIResponseException(PyiCloudException): + """iCloud response exception.""" + def __init__(self, reason, code=None, retry=False): self.reason = reason self.code = code - message = reason + message = reason or "" if code: message += " (%s)" % code + if retry: + message += ". Retrying ..." + + super().__init__(message) - super(PyiCloudAPIResponseError, self).__init__(message) +class PyiCloudServiceNotActivatedException(PyiCloudAPIResponseException): + """iCloud service not activated exception.""" + pass + +# Login class PyiCloudFailedLoginException(PyiCloudException): + """iCloud failed login exception.""" pass -class PyiCloud2SARequiredError(PyiCloudException): - def __init__(self, url): - message = "Two-step authentication required for %s" % url - super(PyiCloud2SARequiredError, self).__init__(message) +class PyiCloud2SARequiredException(PyiCloudException): + """iCloud 2SA required exception.""" + def __init__(self, apple_id): + message = "Two-step authentication required for account: %s" % apple_id + super().__init__(message) -class NoStoredPasswordAvailable(PyiCloudException): +class PyiCloudNoStoredPasswordAvailableException(PyiCloudException): + """iCloud no stored password exception.""" pass -class PyiCloudServiceNotActivatedErrror(PyiCloudAPIResponseError): +# Webservice specific +class PyiCloudNoDevicesException(PyiCloudException): + """iCloud no device exception.""" pass ```

I've not tested these changes in a docker container yet.

wildenrou commented 9 months ago

following - thanks

Junod972 commented 9 months ago

Thanks a lot. Because, all this have a "family" impact. I need to have my dear photos and videos close to me on my NAS. If I can contribute one way or the other I will. Thanks a looooooot. Merci 🤗.

cfurrow commented 9 months ago

There is hope for a fix, and would appreciate others testing if they have a chance: https://github.com/icloud-photos-downloader/icloud_photos_downloader/pull/733

Notes:

many automated tests are broken, and need fixing
there is currently no way to select .com vs .cn domains for icloud (icloud.com is used by default)
there is currently no way to select a photo library, so "All Photos" is used by default

I've had success running a docker container and pulling down all photos/movies from my iCloud photo library:

docker build . -t icloudpd_dev

docker run -it \
  --rm --name icloudpd \
  -v ~/icloudpd-download:/data \
  -e TZ=America/New_York \
  icloudpd_dev \
    icloudpd \
    --directory /data \
    --username my-email@example.com \
    --watch-with-interval 3600

2023-12-09 16:23:42 DEBUG    Authenticating...
iCloud Password:
2023-12-09 16:24:00 INFO     Two-factor authentication is required
Please enter two-factor authentication code: 123456

2023-12-09 16:24:07 WARNING  Failed to parse response with JSON mimetype
2023-12-09 16:24:10 INFO     Great, you're all set up. The script can now be run without user interaction until 2SA expires.
You can set up email notifications for when the two-step authentication expires.
(Use --help to view information about SMTP options.)
2023-12-09 16:24:13 DEBUG    Looking up all photos and videos from album All Photos...
2023-12-09 16:24:13 INFO     Downloading 81324 original photos and videos to /data ...
2023-12-09 16:24:16 DEBUG    Downloading /data/2023/12/09/IMG_1083.MOV...
2023-12-09 16:24:18 INFO     Downloaded /data/2023/12/09/IMG_1083.MOV
2023-12-09 16:24:18 DEBUG    Downloading /data/2023/12/09/IMG_1082.JPG...

bnutzer commented 9 months ago

Hi,

the fix almost works. I have not configured a domain, and the current auth_fix branch fails with this error:

[...]
  File "/home/xxxx/.local/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 393, in _authenticate_with_token
    domain_to_use = req.get('domainToUse')
                    ^^^^^^^
AttributeError: 'Response' object has no attribute 'get'

I have not fully understood what the req.get statement is supposed to do and expect there should be some kind of key check first.

My temporary fix is

        domain_to_use = None
        try:
            domain_to_use = req.get('domainToUse')
        except AttributeError as error:
            print("domainToUse attribute not found.")

With this, I am able to download my icloud photos again 🎉

scaraebeus commented 9 months ago

Hi,

the fix almost works. I have not configured a domain, and the current auth_fix branch fails with this error:
[...]
  File "/home/xxxx/.local/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 393, in _authenticate_with_token
    domain_to_use = req.get('domainToUse')
                    ^^^^^^^
AttributeError: 'Response' object has no attribute 'get'
I have not fully understood what the req.get statement is supposed to do and expect there should be some kind of key check first.

[ . . . ]

I've fixed this in the latest commit. The get function will handle missing keys gracefully, unfortunately, it was being called on an object that doesn't have the method - so even if the key existed, it would still fail with the AttributeError

req.json() does have the get method (as it returns a dictionary), and that is set to self.data just above, so calling self.data.get("domainToUse") should return None if it doesn't exist in the response.

michaelmolino commented 9 months ago

@scaraebeus' fix worked for me. Built a local docker image and successfully grabbed my photos from two accounts! 🎉

Braincoke commented 9 months ago

It didn't work for me at first. I ran it with:

docker run -it --rm -v $(pwd)/mynas/Photos/iCloudBackup:/data -e TZ=America/New_York icloudpd icloudpd --directory /data --username  myemail@mail.com

I had the following error:

pyicloud_ipd.exceptions.PyiCloudNoStoredPasswordAvailableException: No pyicloud password for myemail@mail.com could be found in the system keychain.  Use the `--store-in-keyring` command-line option for storing a password for this username.

But I just needed something quick and dirty to do a backup so I hardcoded my password in base.py and it worked.

cfurrow commented 9 months ago

@Braincoke You need to supply your own email to the docker run command:

docker run -it \
  --rm \
  -v $(pwd)/mynas/Photos/iCloudBackup:/data \
  -e TZ=America/New_York \
  icloudpd icloudpd --directory /data --username  myemail@mail.com

You can see that in the command above, I pass a dummy email, --username myemail@mail.com You would want to replace that with your actual apple id email.

Braincoke commented 9 months ago

Thanks @cfurrow, I did pass my real apple ID to the command.

scaraebeus commented 9 months ago

@Braincoke - I took a quick look and the icloudpd auth flow should prompt for a password if it's not supplied in the command line --password your_password and also not found in the keyring.

With the changes in the pyicloud exception names, this exception isn't being caught by icloudpd to prompt for the password. It was previously looking for pyicloud_ipd.exceptions.NoStoredPasswordAvailable and should now be looking for pyicloud_ipd.exceptions.PyiCloudNoStoredPasswordAvailableException.

I'll review the pyicloud exceptions referenced throughout icloudpd and update.

vw-kombi commented 9 months ago

I am on unraid and hence cant test any of this stuff being reported, but looking forward to testing it once an update goes to the unraid app store.

FaceAce1 commented 9 months ago

遇到了同样问题

winteriscariot commented 9 months ago

The docker solution does not seem to work for me: I'm still getting invalid email/password combo when attempting this via a build docker container:

  560  git clone https://github.com/icloud-photos-downloader/icloud_photos_downloader.git
  561  cd icloud_photos_downloader/
  562  ls
  563  docker build . -t icloudpd_dev
  564  history
  565  docker run -it --rm --name icloudpd -v ~/Pictures/Photos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username my@email.com --watch-with-interval 1800
2023-12-12 07:35:41 DEBUG    Authenticating...
iCloud Password: 
2023-12-12 07:35:44 ERROR    Unknown reason
Traceback (most recent call last):
  File "pyicloud_ipd/base.py", line 220, in authenticate
  File "requests/sessions.py", line 637, in post
  File "pyicloud_ipd/base.py", line 105, in request
  File "pyicloud_ipd/base.py", line 127, in _raise_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "starters/icloudpd_ex.py", line 110, in <module>
  File "starters/icloudpd_ex.py", line 106, in main
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "icloudpd/base.py", line 317, in main
  File "icloudpd/base.py", line 744, in core
  File "icloudpd/authentication.py", line 31, in authenticate_
  File "pyicloud_ipd/base.py", line 204, in __init__
  File "pyicloud_ipd/base.py", line 228, in authenticate
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason'))
[8] Failed to execute script 'icloudpd_ex' due to unhandled exception!
~/.src/icloud_photos_downloader $>

i successfully logged into icloud via browser to confirm that I wasn't fat-fingering my password.

scaraebeus commented 9 months ago

The docker solution does not seem to work for me: I'm still getting invalid email/password combo when attempting this via a build docker container:

The master branch has yet to be updated. One proposed solution is available in PR #734.

lcmartos commented 9 months ago

@winteriscariot I think you're not getting the repo with the fix. The PR that is waiting to merge is https://github.com/scaraebeus/icloud_photos_downloader.git right now.

Also, this don't work for me. I've got the same error:

git clone https://github.com/scaraebeus/icloud_photos_downloader.git

docker build . -t icloudpd_scarabeus --no-cache

docker run -it    --rm --name icloudpd    -v /tmp/icloudpd-download:/data    -e TZ=America/New_York    icloudpd_scarabeus      icloudpd      --directory /data      --username USER@DOMAIN.COM      --watch-with-interval 3600 --password "PASSWORD"
2023-12-12 11:24:26 DEBUG    Authenticating...
2023-12-12 11:24:27 ERROR    Unknown reason
Traceback (most recent call last):
  File "pyicloud_ipd/base.py", line 220, in authenticate
  File "requests/sessions.py", line 637, in post
  File "pyicloud_ipd/base.py", line 105, in request
  File "pyicloud_ipd/base.py", line 127, in _raise_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "starters/icloudpd_ex.py", line 110, in <module>
  File "starters/icloudpd_ex.py", line 106, in main
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "icloudpd/base.py", line 317, in main
  File "icloudpd/base.py", line 744, in core
  File "icloudpd/authentication.py", line 31, in authenticate_
  File "pyicloud_ipd/base.py", line 204, in __init__
  File "pyicloud_ipd/base.py", line 228, in authenticate
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason'))
[7] Failed to execute script 'icloudpd_ex' due to unhandled exception!

michaelmolino commented 9 months ago

@lcmartos git checkout auth_fix

lcmartos commented 9 months ago

@michaelmolino Working! Thank you! I suppose it's just a matter of time before they try and accept the patch.

fasihi01 commented 9 months ago

I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really): wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip mv auth_fix auth_fix.zip unzip auth_fix docker build . -t icloudpd_dev docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username mail@mail.com --watch-with-interval 1800 --password XXX

espro1 commented 9 months ago

I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really): wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip mv auth_fix auth_fix.zip unzip auth_fix docker build . -t icloudpd_dev docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username mail@mail.com --watch-with-interval 1800 --password XXX

I followed these steps and get the error Missing apple_id field:

2023-12-12 16:03:27 DEBUG    Authenticating...
2023-12-12 16:03:28 ERROR    Missing apple_id field
Traceback (most recent call last):
  File "pyicloud_ipd/base.py", line 365, in _authenticate_with_token
  File "requests/sessions.py", line 637, in post
  File "pyicloud_ipd/base.py", line 156, in request
  File "pyicloud_ipd/base.py", line 185, in _raise_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseException: Missing apple_id field

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "starters/icloudpd_ex.py", line 110, in <module>
  File "starters/icloudpd_ex.py", line 106, in main
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "icloudpd/base.py", line 317, in main
  File "icloudpd/base.py", line 744, in core
  File "icloudpd/authentication.py", line 31, in authenticate_
  File "pyicloud_ipd/base.py", line 283, in __init__
  File "pyicloud_ipd/base.py", line 345, in authenticate
  File "pyicloud_ipd/base.py", line 371, in _authenticate_with_token
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid authentication token.', PyiCloudAPIResponseException('Missing apple_id field'))
[7] Failed to execute script 'icloudpd_ex' due to unhandled exception!

I'm sure this is basic user error on my part but I thought I would share in case it isn't.

I also have to pass the password as an option. If I don't, it complains that it isn't in the keyfile.

scaraebeus commented 9 months ago

I followed these steps and get the error Missing apple_id field:

[ . . . ]

I'm sure this is basic user error on my part but I thought I would share in case it isn't.

I also have to pass the password as an option. If I don't, it complains that it isn't in the keyfile.

I'm not sure all of the ways this error can manifest - so this may not apply, however, in some of my testing I was able to get this error if passing an invalid username and/or incorrect password. May want to double check there are no typos there.

wildenrou commented 9 months ago

I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really): wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip mv auth_fix auth_fix.zip unzip auth_fix docker build . -t icloudpd_dev docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username mail@mail.com --watch-with-interval 1800 --password XXX

that worked for me and photos are syncing again. Thanks so much everyon for the great and quick team work on this one. Cheers.

icloud-photos-downloader / icloud_photos_downloader