search

nicohman / rust-wildbow-scraper

Automatically scrapes wildbow's web serials and compiles them into ebooks
GNU General Public License v3.0
95 stars 20 forks source link

paralellization #45

Closed KraXen72 closed 1 year ago

KraXen72 commented 1 year ago

consider implementing paralellization, so multiple requests to chapters get processed at the same time in different threads. currently, it takes several minutes to download the whole book. wasn't the whole point of rust to be """blazingly fast"""?? /j. honestly it's fine even if kept as-is, but would be a cool addition.

nicohman commented 1 year ago

This can't really be done within the current implementation, since it follows 'Next Chapter' links - that's how it includes things like Extra Materials in the right order now. This doesn't make sense, especially given that it caches by chapters now, so you can just download the new chapters as they come out.

KraXen72 commented 1 year ago

i didn't know it cached. that's pretty nice. what happens if there are some edits to the old chapters though?

nicohman commented 1 year ago

In that case, you can delete the cache to force it to re-scrape -- it prints the cache location when run.