Closed biGdada closed 3 years ago
My photostream has around 600 photos in it. I’ll hit this error if I try syncing too many times in a short space of time... I’d guess at about 6 syncs in an hour.
It took days to clear the first time I ran into it. I had sync scheduled every hour an hit the error a lot of times before I noticed it. I think that once you hit the error, every failed attempt adds to the lockout time.
I switched my container off for two days, set it to sync every 6hrs and it’s gone away after that.
yea, i figured out that the lockout time is accumulating. i just wanted to know is there a way to calculate/check the lockout time and maybe modify the script accordingly. for example, before running it - check wherever the account is locked out and, ideally, for how long.... thanks for a fast response 👍
Maybe you're just running the script too often, @biGdada ? I've been running it on a daily cron for some time and did not run into this recently. How often are you running it? Are you still receiving these messages?
If someone is running a script too often, and an error occurs then it’s honestly not good enough coding! You guys got it! Check what cause the error and instead throw a good exception such as, no pictures and exit cleanly. Is that a possibility?
On Thu, Oct 1, 2020 at 4:41 AM menkej notifications@github.com wrote:
Maybe you're just running the script too often, @biGdada https://github.com/biGdada ? I've been running it on a daily cron for some time and did not run into this recently. How often are you running it? Are you still receiving these messages?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ndbroadbent/icloud_photos_downloader/issues/163#issuecomment-702076732, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALE26ZSKPHPHZOOAKF4UXULSIRTGHANCNFSM4QBZIJCQ .
Hi! I have not seen this error myself and I'm just trying to get a handle on this. So please let's move this into a constructive way of communication, so that we can find a solution for this together, ok?
Can you provide more of the output you're getting, so it might be possible to find where this message occurs?
Just set your cron to run every ten minutes and you’ll soon run into it. I’m fairly certain this is due to Apple rate limiting.
I hope my comment was not being none constructive. That is why I said that the coding is not that good and you guys can nail it! Once you know the issue maybe limit number of runs.
Somebody running the script too often is a user problem, not a coding problem.
I was able to reproduce it now and catch the exception. I'll think about a "proper user message" and then check it in.
I re-ran a download 6/7 times right after another and came into this situation. Apple is obviously having some limits implemented here, which makes perfect sense for my. After just a couple of minutes I was able to run the download again for some retries. I guess this is quite a rare situation where "a normal user" would run this script in such short intervals anyways...
@menkej i think you are missing a point here. the throttling depends,among other things, on the amount of photos in your camera roll. if your camera roll has about 5000 pics, you will run into throttling if you run the script once an hour. i was merely asking if anyone knows how this mechanism works in order to calculate the safe frequency of running this script. i dont think a fix is needed.
Well calculation is the fix so you run a command and it just works! Am I missing something?
So we have two different issues here.
For the first issue I have a patch witch I'll put in a pull request later, so that the program exits clearly and informs the user not to run it in short intervals.
For the second issue, I guess from how I read the code that we already have some waiting/retry in the code but maybe the exception should not come out. But I could not reproduce it up till now, as I'm running on a very slow internet connection here (I'm currently in a holiday house). I downloaded some 3k pictures right now without errors. If you could provide the full output of the error you're getting I might be able to figure out where to look. Which command line parameters are you using?
HI @biGdada, I'm still not able to reproduce the behaviour when downloading many files (6k). Are you able to reproduce the error message during downloads? If you can, could you try my fix (see the released pull request). Does it solve the issue for you?
@menkej i couldn't reproduce it as well, the script has been running every hour for about 2 days
i couldn't reproduce it as well, the script has been running every hour for about 2 days
Great, so this should be good now. Thanks for the feedback!
Hey there - I just tried out this yesterday and I've been receiving the dreaded "private db access disabled for this account" error. I've also tried waiting a few hours between attempts and still receive the same error. I have around 24,000 - 25,000 photos and have been trying to use this command:
./icloudpd.py --username <redacted> -p hunter12 --directory ./data --cookie-directory ./cookies --size original --auto-delete --recent 500 --list-albums
@biscuitehh you do --list-albums, meaning no download at all... Did you try without --list-albums?
@menkej yup - I've tried it without --list-albums, --auto-delete, --recent x, and inside the Docker container. No luck no matter what I try/how long I space my requests apart
@menkej just a heads up - it looks like the latest commit that set the default thread count to one fixed my issue. I'll let ya'll know if comes up again :)
I just hit this error again last night... I wasn't doing anything all night so this shouldn't be a rate limiting issue.
The check I do prior to download was successful, but the actual download failed. It fixed itself without any intervention.
It may be possible that this error occurs for other reasons beyond throttling.
2020-10-28 11:01:05 INFO Check 2FA Cookie
2020-10-28 11:01:05 INFO Cookie exists, check expiry date
2020-10-28 11:01:05 INFO Valid two factor authentication cookie found. Days until expiration: 89
2020-10-28 11:01:05 INFO Check download directory mounted correctly
2020-10-28 11:01:19 INFO Check for new files using password stored in keyring...
2020-10-28 11:01:53 INFO Check successful
2020-10-28 11:01:53 INFO No new files detected. Nothing to download
2020-10-28 11:01:53 INFO Web cookie expires: 2020-12-26 @ 22:59:37
2020-10-28 11:01:53 INFO Two factor authentication cookie expires: 2021-01-25 @ 22:59:53
2020-10-28 11:01:53 INFO Next syncronisation at 23:01
2020-10-28 23:01:53 INFO Check 2FA Cookie
2020-10-28 23:01:53 INFO Cookie exists, check expiry date
2020-10-28 23:01:53 INFO Valid two factor authentication cookie found. Days until expiration: 88
2020-10-28 23:01:53 INFO Check download directory mounted correctly
2020-10-28 23:01:53 INFO Check for new files using password stored in keyring...
/home/user/iCloud/2020/IMG_2141.HEIC
2020-10-28 23:02:31 INFO Check successful
2020-10-28 23:02:31 INFO Check detected 1 files requiring download. Verifying list accuracy
2020-10-28 23:02:31 INFO Ignoring 0 files which have already been downloaded
2020-10-28 23:02:31 INFO New files detected: 1
2020-10-28 23:02:31 INFO Starting download of new files for user: user
2020-10-28 23:02:31 INFO Downloading new files using password stored in keyring...
2020-10-28 23:02:32 DEBUG Authenticating...
private db access disabled for this account. Please wait a few minutes then try again. The remote servers might be trying to throttle requests. (ACCESS_DENIED)
2020-10-28 23:02:39 ERROR Error during download - Exit code: 1
2020-10-28 23:02:39 INFO Sending Telegram failure notification
2020-10-28 23:02:39 INFO Telegram failure notification sent successfully
2020-10-28 23:02:39 INFO Web cookie expires: 2020-12-26 @ 22:59:37
2020-10-28 23:02:39 INFO Two factor authentication cookie expires: 2021-01-25 @ 22:59:53
2020-10-28 23:02:39 INFO Next syncronisation at 11:02
2020-10-29 11:02:39 INFO Check 2FA Cookie
2020-10-29 11:02:39 INFO Cookie exists, check expiry date
2020-10-29 11:02:39 INFO Valid two factor authentication cookie found. Days until expiration: 88
2020-10-29 11:02:39 INFO Check download directory mounted correctly
2020-10-29 11:02:39 INFO Check for new files using password stored in keyring...
/home/user/iCloud/2020/IMG_2141.HEIC
2020-10-29 11:03:13 INFO Check successful
2020-10-29 11:03:13 INFO Check detected 1 files requiring download. Verifying list accuracy
2020-10-29 11:03:13 INFO Ignoring 0 files which have already been downloaded
2020-10-29 11:03:13 INFO New files detected: 1
2020-10-29 11:03:13 INFO Starting download of new files for user: user
2020-10-29 11:03:13 INFO Downloading new files using password stored in keyring...
2020-10-29 11:03:13 DEBUG Authenticating...
2020-10-29 11:03:17 DEBUG Looking up all photos and videos from album All Photos...
2020-10-29 11:03:17 INFO Downloading 854 original photos and videos to /home/user/iCloud ...
2020-10-29 11:03:20 INFO Downloading /home/user/iCloud/2020/IMG_2141.HEIC
No concrete explanation, yet. This is a known behaviour that happens from time to time. See also https://github.com/picklepete/pyicloud/issues/194
Maybe some indexing job... Is this reproducible?
Hey all, my lib is roughly 140.000 photos. I never saw one file being downloaded. I always get the error immediately.
"private db access disabled for this account. Please wait a few minutes then try again. The remote servers might be trying to throttle requests. (ACCESS_DENIED)"
I waited hours, days and now 2 weeks. I do use docker and as only env synchronisation_interval=21600 (and tz, user, folder, ...) Is there a way how I can validate the cookie? I once did enter a wrong 2FA code and still got a cookie file?!
Any advice?
Thanks, Ben
I never saw one file being downloaded. I always get the error immediately. @benkrebs can you provide additional information on how you run this program and your python environment, please? Can you see your photos, when you log in into icloud.com with your browser?
Hey @menkej thanks for your fast response. 1.) yes, I can see and browse the photos on iCloud.com:
2.) I am using boredazfcuk/icloudpd dockerized version (latest); running on a local ubuntu server
My config: services: icloudsync: container_name: icloudsync image: boredazfcuk/icloudpd restart: unless-stopped environment:
3.) then I init: docker exec -it icloudsync sync-icloud.sh --Initialise -> what I saw here: even if I provide a wrong 2FA code I get a key and it proceeds to the error message
4.) Python inside the docker:
Hope that helps and thanks!
hi. i'm getting alot of those lately:
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: private db access disabled for this account. Please wait a few minutes then try again. The remote servers might be trying to throttle requests. (ACCESS_DENIED)
with my icloud account. my sons account never gets this error.i suppose thats because my photo library is much bigger. what is the actual limit of requests? is it per hour? is there any info of how this throttling mechanism works?