Open sylvinus opened 8 years ago
Greg mentioned that Internet Archive does "survey crawls" every year with some root zones, but they are pretty spammy, as expected.
Why not starting by indexing only domains from indexed Web pages and doing update when pages are updated? it is not complete, but it is a start.
@thepieuvre currently we reindex everything each month so we can avoid doing updates
However making sure we have the root domain indexed for each page is a good start indeed.
Those spammy URLs are just as valuable as non-spammy URLs.
@indolering yes there are as a dataset, but we don't want to have them in the search results, the index will be large enough as it is!
But wouldn't it help to know which pages those spammy domains link to?
There are a few ranking signals we could extract from data related to the domain name registration and records:
Getting complete whois/zonefile dumps doesn't seem easy at the moment. Any ideas?
A couple interesting links: