Crawler fail - Githubissues

internetarchive / dweb-mirror

Offline Internet Archive project

https://www-dweb-mirror.dev.archive.org/

GNU Affero General Public License v3.0

263 stars 27 forks source link

Crawler fail #203

Closed mitra42 closed 5 years ago

mitra42 commented 5 years ago

Fails on ... https://ia800300.us.archive.org/BookReader/BookReaderJSIA.php?id=themysteryofmary14632gut&itemPath=%2F8%2Fitems%2Fthemysteryofmary14632gut&server=ia800300.us.archive.org&format=json&subPrefix=themysteryofmary14632gut&requestUri=%2Fdetails%2Fthemysteryofmary14632gut With a 404 transport error

mitra42 commented 5 years ago

Diagnosis - its not checking if its a book v Carousel

[x] Add check in CrawlManager > CrawlItem.process before does pages, and get images for carousel instead
[x] check error handling in queue in CrawlManager.process

mitra42 commented 5 years ago

Blocking #202