search

AI-Hypercomputer / JetStream

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).
Apache License 2.0
202 stars 26 forks source link

Add an abstract class for Tokenizer #53

Closed bhavya01 closed 5 months ago

bhavya01 commented 5 months ago

I see that #40 changed the implementation to output only token IDs instead of strings. Will need to update this PR with the new changes.

vipannalla commented 5 months ago

Thanks for the PR, this looks great. I'd appreciate if you can also add unittests (or update existing ones) for the changes.