search

baidu / Familia

A Toolkit for Industrial Topic Modeling
BSD 3-Clause "New" or "Revised" License
2.64k stars 596 forks source link

2个不相干的句子，测得js距离为何是0？应该一样的词才会如此吧 #80

Closed sun7852961 closed 5 years ago

sun7852961 commented 5 years ago

I0124 16:51:52.078809 6469 tokenizer.cpp:73] Vocabulary load successfully! #vocab_size = 294657 Enter Document1: 我是风 Enter Document2: 马和牛 Jensen-Shannon Divergence = 0.0 Hellinger Distance = 0.0

jiangjiajun commented 5 years ago

I0124 16:51:52.078809 6469 tokenizer.cpp:73] Vocabulary load successfully! #vocab_size = 294657 Enter Document1: 我是风 Enter Document2: 马和牛 Jensen-Shannon Divergence = 0.0 Hellinger Distance = 0.0

可以检查下输入的分词结果是怎样的，如果句子分词后所有词均不在词表中，那么是会出现如上情况