search

alasdairforsythe / tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
MIT License
548 stars 19 forks source link

Implemented in the new AI framework Zeta #26

Closed kyegomez closed 10 months ago

kyegomez commented 1 year ago

Hey I like TokenMonster alot and have implemented it into our framework, Zeta, the framework enabling one to build the best multi-modality transformer models!

https://github.com/kyegomez/zeta

#!pip install zetascale

from zeta.tokenizers import TokenMonster
tokenizer = TokenMonster("englishcode-32000-consistent-v1")
tokens = tokenizer.tokenize("Hello world!")
print(tokens)
alasdairforsythe commented 10 months ago

Nice work!