Open sylvinus opened 8 years ago
We should also support the x-robots-tag
http header:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag#using-the-robots-meta-tag
@jhildreth started a branch with some good work that should be merged: https://github.com/commonsearch/cosr-back/compare/master...jhildreth:feature/robots-tag
http://www.robotstxt.org/meta.html
Not sure if Common Crawl already filters those pages, but we should do it on our side too anyway.
Some pointers:
is_indexable
method to DocumentHTMLDocument.is_indexable()
should test for the presence ofnoindex
inself.head_metas["robots"]
nofollow
we could simply test for its presence in close_tag