Closed boredazfcuk closed 9 months ago
I followed these steps and get the error
Missing apple_id field
: [ . . . ] I'm sure this is basic user error on my part but I thought I would share in case it isn't. I also have to pass the password as an option. If I don't, it complains that it isn't in the keyfile.I'm not sure all of the ways this error can manifest - so this may not apply, however, in some of my testing I was able to get this error if passing an invalid username and/or incorrect password. May want to double check there are no typos there.
Thanks for the tip. Just for the record, I've double- and triple-checked and I don't have any typos in username or password.
Thanks for all the work on this everyone. Seems like there are fixes in place for some. I can test too it it gets to the unraid community app store.
I think I found the issue I was having. My password has a special character and if I escape it it works. I should note that I didn't have to do this in 1.16.2.
It doesn't seem to be saving my session, but that's a problem for another day.
Thanks everyone :)
Thanks for all the work on this everyone. Seems like there are fixes in place for some. I can test too it it gets to the unraid community app store.
Yeah I use unraid and I have no idea how to apply the fix lol
Any solution yet?
@amdydesign as above, git checkout auth_fix
. Or wait until branch is merged into master.
I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really): wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip mv auth_fix auth_fix.zip unzip auth_fix docker build . -t icloudpd_dev docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username mail@mail.com --watch-with-interval 1800 --password XXX
how to with this command add --delete-after-download, because its not working
it's multiple commands. your "delete after download" belongs to icloudpd and belongs to the last command:
wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip
mv auth_fix auth_fix.zip
unzip auth_fix
docker build . -t icloudpd_dev
docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username mail@mail.com --watch-with-interval 1800 --password XXX --delete-after-download
I really fine this just above my head, maybe I can do It but I dont want to mess things up, any idea when the unraid image would be ready, and I dont mean an exact time im justing wondering if anyone is working to update that part or is this manual fix is it, this is what we have either use the fix or dont?
it's multiple commands. your "delete after download" belongs to icloudpd and belongs to the last command:
wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip
mv auth_fix auth_fix.zip
unzip auth_fix
docker build . -t icloudpd_dev
docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username mail@mail.com --watch-with-interval 1800 --password XXX --delete-after-download
its downloading photos but didnt delete them after download
it's multiple commands. your "delete after download" belongs to icloudpd and belongs to the last command:
wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip
mv auth_fix auth_fix.zip
unzip auth_fix
docker build . -t icloudpd_dev
docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username mail@mail.com --watch-with-interval 1800 --password XXX --delete-after-download
Works with above for me on Linux and Docker. Great work regarding the workaround! All kudos! But I can't figure out how to download shared libraries with this - is it possible?
I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really): wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip mv auth_fix auth_fix.zip unzip auth_fix docker build . -t icloudpd_dev docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username mail@mail.com --watch-with-interval 1800 --password XXX
Can you elaborate on what exactly to do here? I can't move a file i don't have into a zip file, and so on..
@NGC3982 Here you go: https://github.com/OrZidkani/icloudpd_auth_fix
Make it easier to run.
Big thanks to @scaraebeus
Is there a update for the PIP (windows) Version? I don't use docker. Or do I have to wait for next friday?
Thanks for all the efforts to fix it.
Frank
Does anyone know how to apply this fix for docker running in synology nas?
Is there a update for the PIP (windows) Version? I don't use docker. Or do I have to wait for next friday?
Not sure on timing for the official update. Unfortunately for my fix to be pulled in, there are a few things that need to be in place on my branch for it to be accepted. I've been working through updating the various tests and code coverage and in doing so I am finding some other parts of the pyicloud_ipd/base.py
auth flow that are no longer working correctly due to (possibly) recent changes in how the responses are coming through.
I'm working through it but it will take some time - especially with the holidays coming up.
The only option I'm aware of at this time is to consume the auth_fix branch directly if your environment and setup allows you - with the risk that it has yet to be fully tested and vetted.
@scaraebeus thanks a lot for your hard work and dedication. It is well appreciated by many.
@scaraebeus I join previous comment to thank you for your great contribution here, I patched the main branch with #734, built and installed it with pip and got it working on both raspbian/debian bullseye (11) and bookworm (12) . I am also using edits as described in issue #249 so that I can download both edited media (the ones we see on the phone and in iCloud) and original media in a regular cron backup script to my pi-based NAS, which makes this script critical to me. thanks again and congrats for keeping it working !
Is there a update for the PIP (windows) Version? I don't use docker. Or do I have to wait for next friday?
Not sure on timing for the official update. Unfortunately for my fix to be pulled in, there are a few things that need to be in place on my branch for it to be accepted. I've been working through updating the various tests and code coverage and in doing so I am finding some other parts of the
pyicloud_ipd/base.py
auth flow that are no longer working correctly due to (possibly) recent changes in how the responses are coming through.I'm working through it but it will take some time - especially with the holidays coming up.
The only option I'm aware of at this time is to consume the auth_fix branch directly if your environment and setup allows you - with the risk that it has yet to be fully tested and vetted.
Works on Debian 12 with docker for me until some period of time where the Apple servers either: hang up, as icloudpd appears to idle in the middle of downloading attempt negotiate 2fa and send me an authentication code while icloudpd is in the middle of downloading In both cases there isn't any debug or info logging printed indicating why icloudpd paused, but my network traffic goes to near zero, so I'm guessing the sessions is closed by apple.
Additionally, I'm attempting to download over 180k photos, so that might be why apple is hanging up on me periodically. It doesn't explain the 2fa prompt while icloudpd is running and actively downloading though.
The expected behavior where re-running icloupd picks up where it left off results in diminishing returns: 1st session, 4k downloads. 2nd session: 4k skipped and 2k additional downloads. 3rd session: 6k skipped and 500 additional completed downloads. And so on until no additional downloads complete at all. This might be from pyicloud json handling?
Works! For those using the above solution, a minor change is necessary (I'm on macOS): in the Shell script, change /mnt/iCloud
to $(pwd)
. Then, in Docker Desktop, add the current directory under Settings --> Resources --> File sharing.
ETA: The $(pwd)
comes from this documentation.
Edit 2: Don't use $(pwd)
! The documentation version works because it refers to where you are, but in the script, it cd
s to a temp directory so it would download there. Change it to the directory you want the downloads to go to!
Is saving to the keychain not working for anyone else with the auth_fix branch?
Save password in keyring? [y/N]: y
Two-step authentication required. Your trusted devices are:
0: SMS to ********##
Which device would you like to use? [0]:
Previously I was able to see an option to use a 2FA code instead of SMS
Is saving to the keychain not working for anyone else with the auth_fix branch?
Save password in keyring? [y/N]: y Two-step authentication required. Your trusted devices are: 0: SMS to ********## Which device would you like to use? [0]:
Previously I was able to see an option to use a 2FA code instead of SMS
Steps to reproduce? With the branch I just got a 2FA code on my devices like normal, and it prompted me in the terminal. I don't remember it giving me an option between SMS or 2FA code, though.
Is saving to the keychain not working for anyone else with the auth_fix branch?
Save password in keyring? [y/N]: y Two-step authentication required. Your trusted devices are: 0: SMS to ********## Which device would you like to use? [0]:
Previously I was able to see an option to use a 2FA code instead of SMS
Steps to reproduce? With the branch I just got a 2FA code on my devices like normal, and it prompted me in the terminal. I don't remember it giving me an option between SMS or 2FA code, though.
With icloudpd --username "USERNAME" --password "PASSWORD"
it asks me for a 2FA right away
When I try to save in the keyring it only offers SMS instead of 2FA:
docker run --rm -it --name icloud \
-v /tmp/icloudpd/data:/data \
-v /tmp/icloudpd/cookies:/cookies \
-e "TZ=America/Los_Angeles" \
icloudpd:auth_fix \
icloud --username USERNAME
Enter iCloud password for USERNAME:
Save password in keyring? [y/N]: y
Two-step authentication required. Your trusted devices are:
0: SMS to ********##
Which device would you like to use? [0]
Edit: for now I've just resolved myself to running icloudpd
with a username and password until everything gets sorted out in the main branch
It appears the command line flow for the base pyicloud_ipd
service (which you are invoking with icloud --username USERNAME
) was never updated in the pyicloud - 1.0.0
release to use the new 2FA feature, so it prompts as if it is expecting 2SA (choosing a device to send the code to first).
It also appears, from what I can tell, the icloudpd
flow doesn't offer to save in the keyring directly.
Based on the above, as you've saved the password in the keyring now, try using the icloudpd
directly without the --password
option and it should retrieve from the keyring - and ask for the 2FA without prompting for a device selection.
The option to offer saving in the keyring could be moved into the icloudpd
flow if needed. Not sure if there is already an issue for that or not.
Previous behaviour was that saving to the keychain required SMS MFA, it did not have an option for Apple's built in MFA.
Performing a download would then trigger a second MFA prompt, which had two options for MFA; SMS and Apple.
I noticed this behaviour when attempting to relocate the MFA cookie in my container from $HOME/.local to /config. Took me a lot of deletes/recreates to get it working reliably.
Edit: I think it's also worth mentioning that I actually had two SMS numbers I could use for three of my containers. I have added my phone number as a trusted number to the three accounts of my family members. This allows me to re-authenticate their containers without me needing access to their devices. Very useful feature.
Previous behaviour was that saving to the keychain required SMS MFA, it did not have an option for Apple's built in MFA.
Performing a download would then trigger a second MFA prompt, which had two options for MFA; SMS and Apple.
I noticed this behaviour when attempting to relocate the MFA cookie in my container from $HOME/.local to /config. Took me a lot of deletes/recreates to get it working reliably.
Edit: I think it's also worth mentioning that I actually had two SMS numbers I could use for three of my containers. I have added my phone number as a trusted number to the three accounts of my family members. This allows me to re-authenticate their containers without me needing access to their devices. Very useful feature.
The current pyicloud - 1.0.0
2SA/2FA implementation doesn't appear to take this into account. The command line implementation for pyicloud
doesn't even test for 2FA (hence why it drops straight into asking which device to use). The underlying hooks in the pyicloud
2FA implementation also limit the flexibility for the above use case.
On top of this, there are two separate endpoints used to validate codes depending on whether the 2SA or 2FA flow is triggered. It's possible the endpoints don't really matter and the same endpoint can be used regardless - I'll have to test.
Proposed solution:
icloudpd
if you have a default iDevice and the account is enabled for HSA2 (I believe all accounts are now enabled for HSA2 at this point), prompt for 2FA code directly as it should be auto sent to the deviceicloudpd --list-devices
) could trigger the flow to collect and show the available devices associated with the account, allowing the user to select a different device to send a code to (and displaying the option to bypass this and enter the code if you decide to go with the code auto sent to the default device)Based on the above, as you've saved the password in the keyring now, try using the
icloudpd
directly without the--password
option and it should retrieve from the keyring - and ask for the 2FA without prompting for a device selection.
I think in theory this should work but the code I get via SMS can never be verified successfully:
Enter iCloud password for USERNAME:
Save password in keyring? [y/N]: y
Two-step authentication required. Your trusted devices are:
0: SMS to ********##
Which device would you like to use? [0]: 0
Please enter validation code: ######
Failed to verify verification code
I retract my statement about the pyicloud - 1.0.0
not taking the 2SA/2FA into account, it does, the minimal fix in the auth_fix branch does not include the updated cmdline.py to expose this in the pyicloud_ipd
direct invoking.
@tanookiben - when you attempt to verify the SMS code, have you also already received an auto-generated code to one of your iDevices?
If so, what happens if you enter that code at the prompt instead of the one sent through SMS?
This would at give me a bit of a clue as to where there is something end point related and/or if it's still expecting the first code.
I'm not near my laptop to test, but IIRC, mine crashed at the point where it attempted to list the phone numbers.
Update: All test are now passing for the auth_fix
branch with 99% code coverage (matching the master branch). Hopefully this will allow for the branch to be pulled into the master.
The 2SA/2FA cases noted above will still need to be looked into but I'm hopeful these can be treated as separate issues and allow the core fix to be merged for those that are waiting on this due to various package distribution requirements.
Happy Holidays!
1.17.0 released and has the fix
I've built a new container with iCloud 1.17.0, but the behaviour is not the same. I am no longer presented with the option to perform SMS based multifactor authentication. I also receive a warning about it not being able to parse JSON, which I've not seen before but I'm not too worried about, as the application downloads the photos regardless:
2023-12-20 12:08:01 DEBUG Configure password
2023-12-20 12:08:01 DEBUG Adding password to keyring file: /config/python_keyring/keyring_pass.cfg
2023-12-20 12:08:01 DEBUG Switched to icloudpd: 1.17.0
Enter iCloud password for my@email.com:
Save password in keyring? [y/N]: y
Two-step authentication required.
Please enter validation code
(string) --> 137446
2023-12-20 12:08:46 INFO Starting container initialisation
2023-12-20 12:08:46 DEBUG Generate MFA cookie using password stored in keyring file
2023-12-20 12:08:47 DEBUG Switched to icloudpd: 1.17.0
2023-12-20 12:08:47 ERROR Authentication required for Account. (421)
Please enter two-factor authentication code: 119033
2023-12-20 12:09:20 WARNING Failed to parse response with JSON mimetype
Is SMS based multifactor authentication no longer possible?
Is SMS based multifactor authentication no longer possible?
With the new method used in pyicloud - 1.0.0
it splits the flow between 2SA (using SMS) and 2FA (Auto-sent code) based on whether the iCloud account itself is set for HSA version 2 (it appears all accounts may now be set for this).
The way the base pyicloud
does this check, assuming all accounts are now set as HSA version 2, there is actually no way the 2SA flow would ever be called. I uncovered this awhile ago doing some other testing with an account that doesn't have an iDevice (using the free iCloud service as standalone). In this scenario, I identified a flag noting if the account has a qualifying device. Adding this in to the 2FA check (requiring this to be true) now allows the flow to fall back on the 2SA - but only if there isn't a qualifying device on the account.
I've started working on an enhancement to allow selecting an SMS for 2sa validation. A 2fa code will still be sent to the iDevice automatically, but you should be able to choose an SMS, get a code there, and validate with that instead.
If you want to check that out, see the 2fa_enhancements
branch under scaraebeus/icloud_photos_downloader
- keep in mind this is experimental though. In that branch, you can use the --list-devices
option to present the list of SMS devices and choose to send a code there (or choose to enter the auto sent 2FA code).
It may be worth starting a separate issue for this to be tracked outside of this thread.
I depend on this tool and have been following this issue. Thanks so much for all the hard work and getting this fixed so quickly. You are appreciated! :pray:
I've built a new container with iCloud 1.17.0, but the behaviour is not the same. I am no longer presented with the option to perform SMS based multifactor authentication. I also receive a warning about it not being able to parse JSON, which I've not seen before but I'm not too worried about, as the application downloads the photos regardless:
2023-12-20 12:08:01 DEBUG Configure password 2023-12-20 12:08:01 DEBUG Adding password to keyring file: /config/python_keyring/keyring_pass.cfg 2023-12-20 12:08:01 DEBUG Switched to icloudpd: 1.17.0 Enter iCloud password for my@email.com: Save password in keyring? [y/N]: y Two-step authentication required. Please enter validation code (string) --> 137446 2023-12-20 12:08:46 INFO Starting container initialisation 2023-12-20 12:08:46 DEBUG Generate MFA cookie using password stored in keyring file 2023-12-20 12:08:47 DEBUG Switched to icloudpd: 1.17.0 2023-12-20 12:08:47 ERROR Authentication required for Account. (421) Please enter two-factor authentication code: 119033 2023-12-20 12:09:20 WARNING Failed to parse response with JSON mimetype
Is SMS based multifactor authentication no longer possible?
Can you share the command you used here? I'm able to save credentials to the keyring with icloud --username "USERNAME"
but icloudpd --username "USERNAME"
without a --password
flag always prompts me for a password
Just checking in on this issue, tried the lastest release today and it seems like it is still broken. Was https://github.com/icloud-photos-downloader/icloud_photos_downloader/pull/734, supposed to resolve this issue?
Just checking in on this issue, tried the lastest release today and it seems like it is still broken
If you see the same issue then post here all details. Otherwise open new issue.
Overview
As of about 45mins ago, all four of my containers attempted a download, and all four failed to login.
After removing the keyring file and attempting to recreate it with
icloud --username my@email.com
the password is rejected. I've confirmed the e-mail and password combination is correct.Steps to Reproduce
Expected Behavior
Logs into icloud
Actual Behavior
Fails to login. If password is saved to the keyring, this error is generated:
Context
Seems that Apple may have changed something. Can replicate on Alpine Linux 3.18.3 icloudpd 1.16.2 and Alpine Linux 3.18.5 and icloudpd 1.16.3