We current download the DMOZ data but we only store a boolean signal for the presence of URLs or domains in their dumps.
We should start storing titles and descriptions, and then use them as fallbacks in the search results. An example where this would help is commonsearch/cosr-results#3
We current download the DMOZ data but we only store a boolean signal for the presence of URLs or domains in their dumps.
We should start storing titles and descriptions, and then use them as fallbacks in the search results. An example where this would help is commonsearch/cosr-results#3
We should also add support for
<META NAME="ROBOTS" CONTENT="NOODP">
as explaned here: http://sitemaps.blogspot.com/2006/07/more-control-over-page-snippets.htmlA few pointers:
format_title
andformat_summary
usingurl_metadata
: https://github.com/commonsearch/cosr-back/blob/master/cosrlib/formatting.py