search

boredazfcuk / docker-icloudpd

An Alpine Linux container for the iCloud Photos Downloader command line utility
1.64k stars 156 forks source link

2FA/MFA auth being ignored, seems like container only tries to auth via SMS #366

Closed BackedUpBooty closed 10 months ago

BackedUpBooty commented 1 year ago

I successfully initialized this container a few days ago, with around 34000 photos being downloaded from iCloud to my Synology NAS. Since that first download was completed however the container hasn't been able to pass auth.

Here are the logs for reference:

2023-08-03 22:11:17 DEBUG    Route check to icloud.com successful
2023-08-03 22:11:17 DEBUG    Apple ID: (hidden)
2023-08-03 22:11:17 INFO     Authentication Type: MFA
2023-08-03 22:11:17 DEBUG    Cookie path: /config/(hidden)
2023-08-03 22:11:17 INFO     Cookie expiry notification period: 7
2023-08-03 22:11:17 INFO     Download destination directory: /home/[redacted]/iCloud
2023-08-03 22:11:17 INFO     Folder structure: {:%Y/%m}
2023-08-03 22:11:17 DEBUG    Directory permissions: 750
2023-08-03 22:11:17 DEBUG    File permissions: 640
2023-08-03 22:11:17 INFO     Synchronisation interval: 43200
2023-08-03 22:11:17 INFO     Synchronisation delay (minutes): 0
2023-08-03 22:11:17 INFO     Set EXIF date/time: false
2023-08-03 22:11:17 INFO     Auto delete: false
2023-08-03 22:11:17 INFO     Delete after download: false
2023-08-03 22:11:17 INFO     Photo size: original
2023-08-03 22:11:17 INFO     Single pass mode: false
2023-08-03 22:11:17 INFO     Skip download check: false
2023-08-03 22:11:17 INFO     Skip live photos: false
2023-08-03 22:11:17 INFO     Number of most recently added photos to download: Download All Photos
2023-08-03 22:11:17 INFO     Downloading photos from album: Download All Photos
2023-08-03 22:11:17 INFO     Stop downloading when prexisiting files count is: Download All Photos
2023-08-03 22:11:17 INFO     Live photo size: original
2023-08-03 22:11:17 INFO     Skip videos: false
2023-08-03 22:11:17 INFO     Convert HEIC to JPEG: true
2023-08-03 22:11:17 DEBUG    JPEG conversion quality: 90
2023-08-03 22:11:17 INFO     Pushover notifications enabled
2023-08-03 22:11:17 DEBUG    Notification title: boredazfcuk/iCloudPD
2023-08-03 22:11:17 DEBUG    Pushover user: (hidden)
2023-08-03 22:11:17 DEBUG    Pushover token: (hidden)
2023-08-03 22:11:17 DEBUG    Pushover sound: classical
2023-08-03 22:11:17 INFO     Sending Pushover startup notification
2023-08-03 22:11:18 DEBUG    Pushover startup notification sent successfully
2023-08-03 22:11:18 DEBUG    Download notifications: Enabled
2023-08-03 22:11:18 DEBUG    Delete notifications: Enabled
2023-08-03 22:11:18 INFO     Downloading from: icloud.com
2023-08-03 22:11:18 INFO     Authentication domain: com
2023-08-03 22:11:18 DEBUG    Nextcloud synchronisation trigger: Disabled
2023-08-03 22:11:18 DEBUG    Nextcloud upload: Disabled
2023-08-03 22:11:18 INFO     Ignore Synology extended attribute directories: Disabled
2023-08-03 22:11:18 DEBUG    Group, users:100, already created
2023-08-03 22:11:18 DEBUG    User, Herself:1028, already created
2023-08-03 22:11:18 DEBUG    Set owner and group on icloudpd temp directory
2023-08-03 22:11:18 DEBUG    Set owner and group on config directory
2023-08-03 22:11:18 DEBUG    Set owner and group on keyring directory
2023-08-03 22:11:18 INFO     Directory is writable: /config/python_keyring/
2023-08-03 22:11:18 INFO     Directory is writable: /home/[redacted]/.local/share/
2023-08-03 22:11:18 DEBUG    Configure password
2023-08-03 22:11:18 DEBUG    Using password stored in keyring file: /config/python_keyring/keyring_pass.cfg
2023-08-03 22:11:18 INFO     Check download directory mounted correctly...
2023-08-03 22:11:18 INFO     Failsafe file /home/[redacted]/iCloud/.mounted exists, continuing
2023-08-03 22:11:18 DEBUG    Set owner and group on icloudpd temp directory
2023-08-03 22:11:18 DEBUG    Set owner and group on config directory
2023-08-03 22:11:18 DEBUG    Set owner and group on keyring directory
2023-08-03 22:11:18 INFO     Directory is writable: /config/python_keyring/
2023-08-03 22:11:18 INFO     Directory is writable: /home/[redacted]/.local/share/
2023-08-03 22:11:18 INFO     Keyring file exists, continuing
2023-08-03 22:11:18 DEBUG    Multi-factor authentication cookie exists.
2023-08-03 22:11:18 DEBUG    Valid two factor authentication cookie found. Days until expiration: 86
2023-08-03 22:11:19 DEBUG    Switched to icloudpd: 1.14.5
Aborted!
2023-08-03 22:11:20 INFO     Albums available:
2023-08-03 22:11:20 INFO      - 2023-08-03 22:11:20 INFO     Two-step/two-factor authentication is required!
2023-08-03 22:11:20 INFO      -   0: SMS to *******99
2023-08-03 22:11:20 INFO      -   1: SMS to ********94
2023-08-03 22:11:20 INFO      -   2: Enter two-factor authentication code
2023-08-03 22:11:20 INFO      - Please choose an option: [0]: 
2023-08-03 22:11:20 INFO     Sync user: [redacted]
2023-08-03 22:11:20 INFO     Synchronisation starting at 22:11:20
2023-08-03 22:11:20 INFO     Keyring file exists, continuing
2023-08-03 22:11:20 DEBUG    Check MFA Cookie
2023-08-03 22:11:20 DEBUG    Multi-factor authentication cookie exists.
2023-08-03 22:11:20 DEBUG    Valid two factor authentication cookie found. Days until expiration: 86
2023-08-03 22:11:20 INFO     Check download directory mounted correctly...
2023-08-03 22:11:20 INFO     Failsafe file [redacted].mounted exists, continuing
2023-08-03 22:11:20 INFO     Check for new files using password stored in keyring file
2023-08-03 22:11:20 INFO     Generating list of files in iCloud. This may take a long time if you have a large photo collection. Please be patient. Nothing is being downloaded at this time
2023-08-03 22:11:21 DEBUG    Switched to icloudpd: 1.14.5
  0: SMS to *******xx
  1: SMS to ********xx
  2: Enter two-factor authentication code
Please choose an option: [0]: 2023-08-03 22:11:22 ERROR    Failed check for new files files
2023-08-03 22:11:22 ERROR     - Can you log into icloud.com without receiving pop-up notifications?
2023-08-03 22:11:22 ERROR    Error debugging info:
2023-08-03 22:11:22 ERROR    Aborted!
2023-08-03 22:11:22 ERROR    ***** Please report problems here: https://github.com/boredazfcuk/docker-icloudpd/issues *****
2023-08-03 22:11:22 INFO     Sending Pushover failure notification
2023-08-03 22:11:23 DEBUG    Pushover failure notification sent successfully
2023-08-03 22:11:23 DEBUG    Web cookie exists.
2023-08-03 22:11:23 INFO     Web cookie expires: 2023-10-01 @ 11:49:08
2023-08-03 22:11:23 INFO     Two factor authentication cookie expires: 2023-10-29 @ 08:20:20
2023-08-03 22:11:23 INFO     Days remaining until expiration: 86
2023-08-03 22:11:23 DEBUG    iCloud login counter = 1
2023-08-03 22:11:23 INFO     Synchronisation ended at 22:11:23
2023-08-03 22:11:23 INFO     Total time taken: 00:00:03
2023-08-03 22:11:23 INFO     Next synchronisation at 10:11:20

The CONFIGURATION.md suggests that authentication_type should be set to 2FA (default) or Web, but it always reverts to MFA. Regardless, it's easy to see that an authentication token for MFA and 2FA exists, and that everything is readable, but for some reason it seems to be choosing option 0 for auth and is failing the check for new files.

Any thoughts?

boredazfcuk commented 1 year ago

Thanks, I've updated the guide to replace 2FA with MFA. I've changed it over recently as I feel MFA is a more accurate description.

You just want to initialise your container again. I had the same thing happen last night. Logging in to Apple's server too many times in quick succession seems to trigger an MFA re-authentication. If I restart my container 5 or 10 times in an hour, I get forced to MFA auth again. Same if I set the sync interval to be hourly, which is why I force the sync intervals now with the minimum being 4hrs.

BackedUpBooty commented 1 year ago

So my sync interval is every 12 hours and I don't restart the container at all.

I reinitialized and it was able to check the files for sync, however 12 hours later it failed again (I've included logs from where I reinitialized and then the next attempt to sync, you can see the login count has increased by 1 from 3 to 4, so it's not like I'm spamming it with auth). Any other suggestions for why this might be happening? 

2023-08-04T01:11:20.219059042Z 2023-08-04 10:11:20 INFO     Generating list of files in iCloud. This may take a long time if you have a large photo collection. Please be patient. Nothing is being downloaded at this time
2023-08-04T01:11:20.748379992Z 2023-08-04 10:11:20 DEBUG    Switched to icloudpd: 1.14.5
2023-08-04T01:11:22.342051068Z   0: SMS to ******xxx
2023-08-04T01:11:22.342482558Z   1: SMS to *******xxx
2023-08-04T01:11:22.342535497Z   2: Enter two-factor authentication code
2023-08-04T01:11:22.449106140Z Please choose an option: [0]: 2023-08-04 10:11:22 ERROR    Failed check for new files files
2023-08-04T01:11:22.449669674Z 2023-08-04 10:11:22 ERROR     - Can you log into icloud.com without receiving pop-up notifications?
2023-08-04T01:11:22.450818455Z 2023-08-04 10:11:22 ERROR    Error debugging info:
2023-08-04T01:11:22.452905690Z 2023-08-04 10:11:22 ERROR    Aborted!
2023-08-04T01:11:22.454911838Z 2023-08-04 10:11:22 ERROR    ***** Please report problems here: https://github.com/boredazfcuk/docker-icloudpd/issues *****
2023-08-04T01:11:22.456028694Z 2023-08-04 10:11:22 INFO     Sending Pushover failure notification
2023-08-04T01:11:23.151523901Z 2023-08-04 10:11:23 DEBUG    Pushover failure notification sent successfully
2023-08-04T01:11:23.152356033Z 2023-08-04 10:11:23 DEBUG    Web cookie exists.
2023-08-04T01:11:23.154143666Z 2023-08-04 10:11:23 INFO     Web cookie expires: 2023-10-03 @ 01:11:21
2023-08-04T01:11:23.155026210Z 2023-08-04 10:11:23 INFO     Two factor authentication cookie expires: 2023-10-29 @ 08:20:20
2023-08-04T01:11:23.155796266Z 2023-08-04 10:11:23 INFO     Days remaining until expiration: 85
2023-08-04T01:11:23.156769595Z 2023-08-04 10:11:23 DEBUG    iCloud login counter = 2
2023-08-04T01:11:23.159232923Z 2023-08-04 10:11:23 INFO     Synchronisation ended at 10:11:23
2023-08-04T01:11:23.161030109Z 2023-08-04 10:11:23 INFO     Total time taken: 00:00:03
2023-08-04T01:11:23.162670329Z 2023-08-04 10:11:23 INFO     Next synchronisation at 22:11:20
2023-08-04T13:11:20.166184384Z 2023-08-04 22:11:20 INFO     Synchronisation starting at 22:11:20
2023-08-04T13:11:20.172874960Z 2023-08-04 22:11:20 INFO     Keyring file exists, continuing
2023-08-04T13:11:20.174094727Z 2023-08-04 22:11:20 DEBUG    Check MFA Cookie
2023-08-04T13:11:20.175067682Z 2023-08-04 22:11:20 DEBUG    Multi-factor authentication cookie exists.
2023-08-04T13:11:20.183127093Z 2023-08-04 22:11:20 DEBUG    Valid two factor authentication cookie found. Days until expiration: 89
2023-08-04T13:11:20.184022952Z 2023-08-04 22:11:20 INFO     Check download directory mounted correctly...
2023-08-04T13:11:20.239116085Z 2023-08-04 22:11:20 INFO     Failsafe file /home/[redacted]/iCloud/.mounted exists, continuing
2023-08-04T13:11:20.241926147Z 2023-08-04 22:11:20 INFO     Check for new files using password stored in keyring file
2023-08-04T13:11:20.242744073Z 2023-08-04 22:11:20 INFO     Generating list of files in iCloud. This may take a long time if you have a large photo collection. Please be patient. Nothing is being downloaded at this time
2023-08-04T13:11:20.797455431Z 2023-08-04 22:11:20 DEBUG    Switched to icloudpd: 1.14.5
2023-08-04T13:25:58.423203126Z 2023-08-04 22:25:58 INFO     Check successful
2023-08-04T13:25:58.424568188Z 2023-08-04 22:25:58 INFO     No new files detected. Nothing to download
2023-08-04T13:25:58.425390213Z 2023-08-04 22:25:58 DEBUG    Web cookie exists.
2023-08-04T13:25:58.427195662Z 2023-08-04 22:25:58 INFO     Web cookie expires: 2023-10-03 @ 04:45:39
2023-08-04T13:25:58.428033598Z 2023-08-04 22:25:58 INFO     Two factor authentication cookie expires: 2023-11-02 @ 04:46:07
2023-08-04T13:25:58.428833285Z 2023-08-04 22:25:58 INFO     Days remaining until expiration: 89
2023-08-04T13:25:58.429681586Z 2023-08-04 22:25:58 DEBUG    iCloud login counter = 3
2023-08-04T13:25:58.432123332Z 2023-08-04 22:25:58 INFO     Synchronisation ended at 22:25:58
2023-08-04T13:25:58.433815943Z 2023-08-04 22:25:58 INFO     Total time taken: 00:14:38
2023-08-04T13:25:58.435635275Z 2023-08-04 22:25:58 INFO     Next synchronisation at 10:11:20
2023-08-05T01:11:20.438957839Z 2023-08-05 10:11:20 INFO     Synchronisation starting at 10:11:20
2023-08-05T01:11:20.444476643Z 2023-08-05 10:11:20 INFO     Keyring file exists, continuing
2023-08-05T01:11:20.445317334Z 2023-08-05 10:11:20 DEBUG    Check MFA Cookie
2023-08-05T01:11:20.446499734Z 2023-08-05 10:11:20 DEBUG    Multi-factor authentication cookie exists.
2023-08-05T01:11:20.451972477Z 2023-08-05 10:11:20 DEBUG    Valid two factor authentication cookie found. Days until expiration: 88
2023-08-05T01:11:20.452837006Z 2023-08-05 10:11:20 INFO     Check download directory mounted correctly...
2023-08-05T01:11:20.471028882Z 2023-08-05 10:11:20 INFO     Failsafe file /home/[redacted]/iCloud/.mounted exists, continuing
2023-08-05T01:11:20.474118022Z 2023-08-05 10:11:20 INFO     Check for new files using password stored in keyring file
2023-08-05T01:11:20.475055599Z 2023-08-05 10:11:20 INFO     Generating list of files in iCloud. This may take a long time if you have a large photo collection. Please be patient. Nothing is being downloaded at this time
2023-08-05T01:11:21.100336537Z 2023-08-05 10:11:21 DEBUG    Switched to icloudpd: 1.14.5
2023-08-05T01:11:23.027628359Z   0: SMS to ******xxx
2023-08-05T01:11:23.028038488Z   1: SMS to *******xxx
2023-08-05T01:11:23.028081469Z   2: Enter two-factor authentication code
2023-08-05T01:11:23.110328275Z Please choose an option: [0]: 2023-08-05 10:11:23 ERROR    Failed check for new files files
2023-08-05T01:11:23.111461275Z 2023-08-05 10:11:23 ERROR     - Can you log into icloud.com without receiving pop-up notifications?
2023-08-05T01:11:23.112593694Z 2023-08-05 10:11:23 ERROR    Error debugging info:
2023-08-05T01:11:23.114221091Z 2023-08-05 10:11:23 ERROR    Aborted!
2023-08-05T01:11:23.115044705Z 2023-08-05 10:11:23 ERROR    ***** Please report problems here: https://github.com/boredazfcuk/docker-icloudpd/issues *****
2023-08-05T01:11:23.116020133Z 2023-08-05 10:11:23 INFO     Sending Pushover failure notification
2023-08-05T01:11:24.140268753Z 2023-08-05 10:11:24 DEBUG    Pushover failure notification sent successfully
2023-08-05T01:11:24.141601606Z 2023-08-05 10:11:24 DEBUG    Web cookie exists.
2023-08-05T01:11:24.143485479Z 2023-08-05 10:11:24 INFO     Web cookie expires: 2023-10-04 @ 01:11:22
2023-08-05T01:11:24.144383867Z 2023-08-05 10:11:24 INFO     Two factor authentication cookie expires: 2023-11-02 @ 04:46:07
2023-08-05T01:11:24.145227147Z 2023-08-05 10:11:24 INFO     Days remaining until expiration: 88
2023-08-05T01:11:24.146083410Z 2023-08-05 10:11:24 DEBUG    iCloud login counter = 4
2023-08-05T01:11:24.148526221Z 2023-08-05 10:11:24 INFO     Synchronisation ended at 10:11:24
2023-08-05T01:11:24.150187099Z 2023-08-05 10:11:24 INFO     Total time taken: 00:00:04
2023-08-05T01:11:24.151817701Z 2023-08-05 10:11:24 INFO     Next synchronisation at 22:11:20
boredazfcuk commented 1 year ago

Try setting skip_check=true.

Tye download check has issues on some larger libraries, so disable it and see if it helps.

BackedUpBooty commented 1 year ago

Thanks. Made the change, didn't work on just a restart, I reinitialized and it went through the whole photo library saying the photos already existed, and then it did exactly the same 12 hours later (which is when I set it for). skip_check is definitely set to true, so not sure why it's not doing a full library check

boredazfcuk commented 1 year ago

When a check runs, it goes through your entire library and checks to see if there are any files missing. If there are not, the script waits until the next cycle. If there are, it kicks off a download run, where it re-scans the entire library, downloading the missing files.

Both the check and the download run will cause it to authenticate against Apple's servers, and these two log-ins in quick succession can cause Apple to trigger and MFA authentication request.

Disabling the check means it only logs in once per run, so can help because it doesn't do the double authentication.

I've never seen this behaviour where MFA is required upon every login. I'm wondering if this is a configurable option in Apple's icloud.com... to require MFA on every auth. If it is, then disabling this will solve your issue.

boredazfcuk commented 1 year ago

Well, I've had a look through the settings on my AppleID and can't see any such option, so that can't be it.

boredazfcuk commented 10 months ago 
2023-08-04T01:11:22.342051068Z   0: SMS to ******xxx
2023-08-04T01:11:22.342482558Z   1: SMS to *******xxx
2023-08-04T01:11:22.342535497Z   2: Enter two-factor authentication code
2023-08-04T01:11:22.449106140Z Please choose an option: [0]: 2023-08-04 10:11:22 ERROR    Failed check for new files files

Whenever you see this in the logs, you need to re-initialise the container. This happens to me a lot if I restart the container too many times. Each time you do, the container authenticates against Apple's servers. Do this a lot in quick succession and Apple sees it as suspicious behaviour and prevents login until MFA is reconfirmed.