Open joonaspaakko opened 1 year ago
I will also add that I'm been seeing this more and more as my Audible library has grown. If I attempt to do a "full" extraction now, it is 100% impossible. It generally takes me 5-10 tries to just get the library. Then another 5-10 tries to add the collections. Then another 5-10 tries to go back and add the ISBN numbers.
I suspect that the Audible firewall is seeing the huge number of page requests as an attack, and locking down the source IP for a limited time.
My library now has over 1,700 purchased books, so it ends up being a huge number of "hits" against the audible site. Your suggestion of lowering the extraction rate would be slower, but it is definitely worth it to avoid tripping the audible firewall lockout. :-)
P.S. -- This widget is awesome. I only found out about it a couple of months ago, and I've been a daily user of it on my phone, to sort the series (next book). It became especially useful after Audible removed that feature from their own player app, for some unknown (and probably unholy) reason. THANK YOU for this extension.
-Keith
@wizardfish2, are you using the current official release 0.2.8 or the v.0.2.9 draft that is in the GitHub repository? I updated it just now. It's still a bit unfinished, but you might have more luck with it.
This issue is actually still a bit of a mystery in terms of what the limits are. Sometimes it feels like throttling the requests doesn't really matter, it blocks you anyways... and other times it goes through without a hitch. The weirdest thing is that when I'm using the Brave browser it always stops at the tail end of my series extraction. But in a plain old Chrome installation using same amount of throttling, it goes through fine.
I have/had been using the v.0.2.8 version. I just loaded the v.0.2.9 draft and am running that now. It is definitely getting farther (on a full update) than 0.2.8 could get in a single pass.
Will update progress shortly.
UPDATE: Nice! I did the full update (Lib + Collections + ISBN) in a single pass, with no failures. It took quite a bit longer, at around 40 minutes, but that IS PERFECTLY FINE since it actually worked all the way through.
I also will mention that the new "extension tools" pulldown is very convenient.
Thanks for this -- it's awesome!
Wow... 40 minutes is super slow. Was ISBN extraction most of that?
ISBN extraction is always going to be super slow due to rate limiting in the Google API. But yea basically if it gets that far, it's going to be fine because it's done extracting stuff from Audible.
Yes, ISBN was probably 75% of the time. I'm doing a couple of additional runs (full and incremental), and so far Audible has NOT locked me out for the 10 minute timeout. Did I say "very nice!" yet? Because very nice. :-)
The incremental re-downloads took about 3 minutes for Library + Collections... and then 15 minutes for the ISBN re-check. So yeah, you're certainly right about the ISBN limiting.
Just for informational purposes, I am on a 1 GB WAN feed, which also backs up your statement about Google API being the limiter, rather than network.
Good :)
I did an extraction test as well, where I extracted everything but ISBNs:
859
books 2 minutes
(Doesn't extract data for series)1135
books ~6 minutes
That isn't fast by any means, but I guess I can live with that if I have to.
The ISBN handling in the extension could be better though. I've kinda neglected it... If you need to do a full extraction, which happens... You always have to throw away previously extracted ISBNs but that could be avoided.
Just a few ideas for myself:
To avoid the issue where removing library data gets rid of ISBNs it could pick up all ISBNs when the user removes library data and store them separately until the next time ISBNs are extracted
Maybe it would be helpful if you could import just the ISBN numbers from a previous raw data export
, so if you did delete ISBNs for whatever reason, you could bring them back without having to do an extraction (or at least an extraction that takes super long...).
Right now after you've extracted ISBNs once, the partial extraction tries to find ISBNs for any books that don't have them already...
This is just an opinion from the peanut gallery, but I think your last option is probably the most reasonable. It will work for 99% of the cases (either no new work / ISBN's required or they aren't going to be found on later attempts anyway).
And for the 1% -- those who KNOW an ISBN is available, or know what it is already -- have two options.
In my opinion, it wouldn't make sense to spend a huge amount of time on something that few will need/want at all, especially if there is already the roundabout way to deal with it.
Reopening because I seem to be getting this again...
Only when doing a full extraction (removing all data and extracting). While updating existing data it seems to pull through just fine.
Extraction may seem to freeze and afterwards you'll notice that any page in Audible is blocked with the message
Whoops. We're not sure what happened, but something went wrong.
:Sometimes the extraction does push through eventually, but even if it does, it will be missing some or a lot of data, because the issue is that for some reason Audible started blocking your access and the extractor can't do its thing. It may also stop the extraction entirely.
If you wait for about 10 minutes you should be able to access audible again. Although it might be pointless to try the extraction again as it's likely to repeat the issue.