arzkar / ao3-cli

A CLI to download from archiveofourown.org using their built-in download option.
Apache License 2.0
9 stars 0 forks source link

Add a rety-after to cool down once an user hits AO3's rate limit #4

Open arzkar opened 1 year ago

arzkar commented 1 year ago

Possibly. Can you share the fic url in which you are getting this error. I will try to find a fix.

This isn't a situation where a specific URL triggers it at all. It gets triggered after downloading anywhere from 30-50 fics in a row, after which every URL for a good long while will give "Fanfiction not found". Eventually it'll escape that and get back to working (whenever my IP gets out of AO3 jail) but until then, it doesn't matter which URL I try; they'll all fail. (If I attempt to browse to one in my browser at that moment, it'll give me the "Retry later" message, too.)

Another AO3-related script had this about it: "Try to keep your ao3 browsing to a minimum while the script is running. It won't break anything, but it may cause you to hit ao3's limit on how many hits to the site you are allowed within a certain time frame. This limit is per user, or per IP if you are not logged in. If this happens, the script will pause for 5 minutes to let the limit reset, and you may see a "Retry later" message when you try to open an ao3 page during that time. Don't be alarmed by this, just wait it out."

You can find more information on this error in the issue for this script here: https://github.com/radiolarian/AO3Scraper/issues/24 They note it's a 429 error. You'll notice the previous script (whence I copied the info from) said it was designed to pause for five minutes to wait out the error - is that something that could be included in your script?

This script works great for one or two files here and there, but for a large list of them (I'm attempting to grab 20,000+ fics for a friend, the ids of which I grabbed using that AO3Scraper above), it's going to run into AO3's limits on usage over and over.

Originally posted by @Kyther in https://github.com/arzkar/ao3-cli/issues/2#issuecomment-1000642398