panic: assignment to entry in nil map

alasdairforsythe / tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

MIT License

528 stars 20 forks source link

panic: assignment to entry in nil map #11

Closed botsbreeder closed 1 year ago

botsbreeder commented 1 year ago

Thank you for your work.

I tried to train vocab with a new code but it is failing

Loading 3.dict
2023/07/03 21:07:33 Parsing 3.special.json
Charset: UTF-8, Capcode Enabled
Optimization mode: 4 (strict)
Vocabulary size: 65536
Single byte tokens: 233
Loading normalized.tsv
panic: assignment to entry in nil map

goroutine 1 [running]:
main.main()
        trainvocab.go:1551 +0x1d33

alasdairforsythe commented 1 year ago

Thanks for reporting the bug. I've fixed it and recompiled the binaries.