search

p0ody / ff2ebook

WIP.
http://www.ff2ebook.com
18 stars 2 forks source link

error downloading fanfics again #60

Closed kolbdog3333 closed 2 years ago

kolbdog3333 commented 2 years ago

When downloading fanfics it stops at 20 chapters then it just says Fetching chapter 32 2/5 Error caching chapter and just continues then just fails after 100 fetches

p0ody commented 2 years ago

Seems like cloudflare is being more of a pain once in a while, the scraper is having difficulty.

You might want to try again later.

image

bastien8060 commented 2 years ago

im suggesting something, I think chapters should be cached. not the stories which have been cached (archived) since a long time ago. I don't know if this is already implemented, but I know for sure it wasn't. Maybe a cache at the scraper level could reduce the number of requests sent to cf.

p0ody commented 2 years ago

This would be a good idea, but the thing is there is no way AFAIK to know if a chapter has been modified since it has been cached.

The only thing that says a story has been modified is the Updated date, but it could be anywhere in the story.

image

So if the Updated date change, it would need to update all the chapter anyway, which is pretty much what it is doing already.

BTW, i updated the scraper to wait on these new cloudflare page instead of closing ASAP and retrying, so it should have fixed the problem.