go-ego / gse

Go efficient multilingual NLP and text segmentation; support English, Chinese, Japanese and others.
Apache License 2.0
2.57k stars 215 forks source link

gse dict lose efficacy when set an empty string #172

Open CharlieWangKD opened 1 year ago

CharlieWangKD commented 1 year ago
func main() {
    seg := new(gse.Segmenter)
    seg.Dict = gse.NewDict()
    seg.Init()
    seg.AddToken("bj", 100, "n")
    fmt.Println(seg.Dictionary())
    fmt.Println(seg.Find("bj"))
    seg.AddToken("", 100, "n")
    fmt.Println(seg.Dictionary())
    fmt.Println(seg.Find("bj"))
}
// output:
// &{0xc000140000 1 [{[[98 106]] 100 n 0 []}] 100}
// 100 n true
// &{0xc000140000 1 [{[[98 106]] 100 n 0 []} {[] 100 n 0 []}] 200}
// 0  false

Description

The Find func of dict will lose efficacy when I set an empty string, the instance will be unable to cutting string.