Update C# tokenizer libraries - Githubissues

openai / openai-cookbook

Examples and guides for using the OpenAI API

https://cookbook.openai.com

MIT License

59.99k stars 9.56k forks source link

Update C# tokenizer libraries #1535

Open luisquintanilla opened 3 weeks ago

luisquintanilla commented 3 weeks ago

Summary

This PR:

[x] Adds Microsoft.ML.Tokenizer library to C# libraries
[x] Removes deprecated SharpToken library.
[x] Removes unmaintained GPTTokenizer library.

Motivation

Microsoft.ML.Tokenizers supplants deprecated / unmaintained libraries. It's also the recommended library for .NET and maintained by Microsoft.

Source: https://github.com/dotnet/machinelearning/tree/main/src/Microsoft.ML.Tokenizers Package: https://www.nuget.org/packages/Microsoft.ML.Tokenizers

luisquintanilla commented 3 weeks ago

cc: @tarekgh