search

Genius / trackback_scraper

116 stars 12 forks source link

Seriously... #2

Closed rjonesx closed 10 years ago

rjonesx commented 10 years ago

First, I am glad to see Rap Genius back in the results, you deserve to be there because you are the best site in the industry - period. Google should care about that more than anything else...

However.... A trackback scraper? The technology employed by the largest link removal services out there (Rmoov, remove'em, link detox, link risk, etc.) outpaces this type of tech by literally years, but sites using their platforms take months to recover. Google often takes weeks simply to read a reconsideration request, much less pull a penalty. And that doesn't take into account the time necessary to actually get links removed in sufficient quantities to make Google happy.

a-warner commented 10 years ago

Thank you for your feedback, (and it's definitely good to list alternatives to this method of scraping) but this is a code repository, so I'm not sure it's the best forum for this type of discussion. If you want to submit a pull to allow for faster/more efficient/more accurate scraping, that would be great though! I'd love for this tool to eventually become generalized, but for now it's pretty specific to our use case.