Support Vertex AI's PaLM models

knuddelsgmbh / jtokkit

JTokkit is a Java tokenizer library designed for use with OpenAI models.

https://jtokkit.knuddels.de/

MIT License

516 stars 38 forks source link

Support Vertex AI's PaLM models #58

Closed glaforge closed 6 months ago

glaforge commented 9 months ago

It would be great if jtokkit went beyond just OpenAI models and also supported other models. In particular, I'm interested in the support of the PaLM model from Google. Would you consider expanding to other models?

tox-p commented 9 months ago

There are currently no plans to support other encodings