Importing wikidata would be (for starters) a good way to associate a lot of official URLs to their named entity and wikipedia page. This would help commonsearch/cosr-results#4 for instance.
The way we import Alexa should be a good starting point.
For a first version I think it should be ok to store (key, value) in rocksdb as (normalized_url, (name, english description, english wikipedia slug)).
https://www.wikidata.org/wiki/Wikidata:Database_download
Importing wikidata would be (for starters) a good way to associate a lot of official URLs to their named entity and wikipedia page. This would help commonsearch/cosr-results#4 for instance.
The way we import Alexa should be a good starting point.
For a first version I think it should be ok to store (key, value) in rocksdb as (normalized_url, (name, english description, english wikipedia slug)).
Wikidata should then be added to https://about.commonsearch.org/data-sources