Open sylvinus opened 8 years ago
First step is done, now we split words in the domain name from a vocabulary extracted from the title and summary.
Next step would be to use a much larger vocabulary (include the body? use norvig.com/ngrams/count_1w.txt ?), and improve the splitting itself, probably with statistical methods?
Not sure if this should be done before indexing or completely in Elasticsearch, but it would be helpful for cases like commonsearch/cosr-results#5 to improve the tokenization of URLs to allow better partial matches.
As a first step, could the presence of separated terms ("Le Monde") in the title have an influence on the tokenization of the URLs? ("lemonde.fr" => "le monde fr" in addition to "lemonde fr")