search

togethercomputer / stripedhyena

Repository for StripedHyena, a state-of-the-art beyond Transformer architecture
Apache License 2.0
299 stars 21 forks source link

Handle char level tokenization #11

Closed brianhie closed 9 months ago

brianhie commented 9 months ago

Add char level tokenizer, modify generation code to handle batched inference with char tok

Zymrael commented 9 months ago

Also run black and isort!

brianhie commented 9 months ago

Also run black and isort!

Done 😄