Open sylvinus opened 8 years ago
Sebastian from Common Crawl just did a very interesting first pass on spam in the dumps: https://gist.github.com/sebastian-nagel/beb244bf1f7092a06a1479335a5e268b
This script is able to detect a few webspam clusters based on their domain name and pagerank similarity.
As explained in our blog post, our host-level PageRank is very experimental and still very subject to spam.
Here is a list of our current ideas to improve it, feel free to contribute yours!
rel=nofollow
linksGoing to URL-level PageRanks would obviously help a a lot but it is out of scope for this issue.