Improve host-level PageRanks

[sylvinus]

As explained in our blog post, our host-level PageRank is very experimental and still very subject to spam.

Here is a list of our current ideas to improve it, feel free to contribute yours!

• [x] Don't follow rel=nofollow links
• [ ] Better weights on the edges (treat links between subdomains differently? give less weight for links in the boilerplate and/or at the end of the page? give more weight depending on the number of distinct pages linking to the domain?)
• [ ] Try to group domains belonging to the same owner (By IP address/DNS info? See #15)

Going to URL-level PageRanks would obviously help a a lot but it is out of scope for this issue.

[sylvinus]

Sebastian from Common Crawl just did a very interesting first pass on spam in the dumps: https://gist.github.com/sebastian-nagel/beb244bf1f7092a06a1479335a5e268b

This script is able to detect a few webspam clusters based on their domain name and pagerank similarity.