Open mitra42 opened 5 years ago
There were two sets of throttling - Haproxy (Sam) and then Gio. Haproxy should be open now, which means will get an error from Gio if go over. We need to spot that error and requeue (in the crawler)
STR
mv /Volumes/x/archiveorg /Volumes/x/archiveorg-
internetarchive -sc # in debugger
rm /Volumes/x/archiveorg/*/*related*
internetarchive -sc # in debugger
(if its doing a lot of thumbnail fetches then know that a 504 hit previously now worked, so repeat
Note - I couldnt repeat with these steps, might need to do on higher speed link at IA in which case maybe not an issue
When fetching related items during a crawl, its getting 429 (?) errors, This is Gio's API throttling (reasonably) to protect the search. I should throw these into a queue or otherwise reschedule them.