search

Embedding / Chinese-Word-Vectors

100+ Chinese Word Vectors 上百种预训练中文词向量
Apache License 2.0
11.82k stars 2.32k forks source link

加载词向量时出现大量警告 #58

Closed yewenjunfighting closed 5 years ago

yewenjunfighting commented 5 years ago

你好，想问下加载时为什么会有‘: duplicate word’的警告 2019-02-17 18:40:35,354 : INFO : loading projection weights from ../sgns.financial.word 2019-02-17 18:42:05,926 : WARNING : duplicate word '..................................................................................................' in ../sgns.financial.word, ignoring all but first 2019-02-17 18:42:10,105 : WARNING : duplicate word '..................................................................................................' in ../sgns.financial.word, ignoring all but first

shenshen-hungry commented 5 years ago

这个是因为'..................................................................................................'在分词的时候被认为是一个词，但是长度超过了最大长度，所以形如这样的都被截断了，才造成了重复。实际上这些词没什么意义，可以忽略这些重复的。

yewenjunfighting commented 5 years ago

👌了解，谢谢解答