Update tokenmonster.py - Githubissues

alasdairforsythe / tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

MIT License

548 stars 19 forks source link

Update tokenmonster.py #28

Open amazingvince opened 11 months ago

amazingvince commented 11 months ago

Fix bug in changing size of tokenizer with special tokens. This function returns none instead of the data object that its updating. This cause it to break.