search

go-ego / gse

Go efficient multilingual NLP and text segmentation; support English, Chinese, Japanese and others.
Apache License 2.0
2.57k stars 215 forks source link

Why dict.totalFreq not reduce freq in RemoveToken #191

Open zmq412 opened 3 weeks ago

zmq412 commented 3 weeks ago

if doing AddToken and RemoveToken a same word repeatedly，the totalFreq will increasingly larger。

zhaomoqiang commented 2 weeks ago

// AddToken add a token to the dictionary func (dict *Dictionary) AddToken(token Token) error { ....... dict.totalFreq += token.freq ...... return nil }

// RemoveToken remove token in dictionary func (dict *Dictionary) RemoveToken(token Token) error { bytes := textSliceToBytes(token.text) return dict.trie.Delete(bytes) }