commonsearch / cosr-back

Backend of Common Search. Analyses webpages and sends them to the index.

https://about.commonsearch.org

Apache License 2.0

122 stars 24 forks source link

Add first document-level quality signals #28

Open sylvinus opened 8 years ago

sylvinus commented 8 years ago

We will need to have a model that evaluates many features from documents and gives us a document quality score.

Before doing any machine learning, it would be great to explore the first few features/signals we could include.

A first list of ideas, please add your own!

Vocabulary (we should look at email spam filters)
Broken HTML (this is rather broad!)
Use of tags like <blink> :-)
Usage of known JavaScript trackers/libraries (could be good or bad)
Specific services like domain parking
Usage of ALL CAPS text?

sylvinus commented 8 years ago

34 may give us other signal ideas

IvRRimum commented 8 years ago

404 pages ?

indolering commented 7 years ago

Responsive design
Accessible design
JS modals
Number of redirects required to get to page
Page load time
Total number of requests to load page
URL appearing in sitemap file
Parse URL, subdomains, and query parameters into arrays?
Domain
TLD (normalized using public suffix list)
Authority ranking of outbound links
Broken outbound links
Use of HTTPS
Use of microdata formatting?