search

dmitry-brazhenko / SharpToken

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
https://www.nuget.org/packages/SharpToken
MIT License
214 stars 14 forks source link

cohere support #38

Open omri-suissa-clearmash opened 7 months ago

omri-suissa-clearmash commented 7 months ago

Cohere has two tokenizers: chat embeddings

It would be great to add them to this project.