search

FanfictionBot / reddit-ffn-bot

A reddit auto reply bot.
53 stars 18 forks source link

Incorrect links generated by linksub() function #95

Closed moonpotato closed 8 years ago

moonpotato commented 8 years ago

When using the linksub function, it looks like it is using the old download links for the fics instead of generating a current version. This poses a problem because of #92. The links being output are no longer valid, pointing to the expired www.p0ody-files.com domain.

tusing commented 8 years ago

Thanks. Fixing this should be a simple find-replace. I'll get to it soon.

moonpotato commented 8 years ago

(I haven't read the new code fully yet) does this mean that the bot retrieves the data from the old posts instead of re-scraping it from the story pages?

tusing commented 8 years ago

Indeed. Originally, I implemented linksub(...) by re-scraping. A few problems arose:

  1. Re-scraping every story for every thread took up far too much time. We have randomized delays to obfuscate the bot from seeming like a bot, and a single linksub call could run for minutes depending on how many threads you linked.
  2. Re-scraping so many stories at once actually got the bot IP blocked from fanfiction.net during my testing. I had to spin up a new instance with a different IP. Sure, I could add a story limit, but it's still risky.

So we trade updated fic information for much more speed and stability. Because the bot doesn't need to scrape fanfiction websites anymore for thread recommendations, you can link as many threads as you want and still expect a speedy reply - speedier, in fact, than an actual linkffn(...) call. All with absolutely no risk of getting the bot banned from sites like ff.net.

moonpotato commented 8 years ago

Understood. Yeah, I figured it was something like that.