Update tokenmonster.py - Githubissues

alasdairforsythe / tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

MIT License

551 stars 21 forks source link

Update tokenmonster.py #28

Open amazingvince opened 1 year ago

amazingvince commented 1 year ago

Fix bug in changing size of tokenizer with special tokens. This function returns none instead of the data object that its updating. This cause it to break.