search

dmitry-brazhenko / SharpToken

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
https://www.nuget.org/packages/SharpToken
MIT License
214 stars 14 forks source link

Model prefix encodings #22

Closed ian-cameron closed 1 year ago

ian-cameron commented 1 year ago

21