SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
214
stars
14
forks
source link
Add pointer to Microsoft.ML.Tokenizers #37
Closed
ericstj closed 7 months ago
Add benchmark info for Microsoft.ML.Tokenizers and direct folks to this package.
cc @dmitry-brazhenko @tarekgh @luisquintanilla