search

dmitry-brazhenko / SharpToken

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
https://www.nuget.org/packages/SharpToken
MIT License
214 stars 14 forks source link

SharpToken.GptEncoding.GetEncoding(o200k_base) takes minutes #45

Open vovanb opened 4 months ago

vovanb commented 4 months ago

Hi I am using your library in MAUI android. For some reason method SharpToken.GptEncoding.GetEncoding(o200k_base) first time takes minutes to respond What can be issue?

dmitry-brazhenko commented 4 months ago

Hey!

Sounds interesting. Could you share some more details please

  1. Text length
  2. android ver
  3. device

Probably could you share apk to check it? My guess is that it takes significant time just to read tokenizer dict, but not sure about it.