TikToken.EncodingForModel("gpt-3.5-turbo") hangs when context is a WPF appplication.

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

11.61k stars 784 forks source link

TikToken.EncodingForModel("gpt-3.5-turbo") hangs when context is a WPF appplication. #175

Closed felipedrumond closed 1 year ago

felipedrumond commented 1 year ago

I have a WPF C# application that uses TikToken tikToken = TikToken.EncodingForModel("gpt-3.5-turbo"); and will hand, even after getting a 200 http status when fetching https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken

Using my same library in a Console context doesn't hang. 1) what can be happening in the WPF context? 2) is there a way to bypass the download of cl100k_base.tiktoken all the time?

hauntsaninja commented 1 year ago

I don't know what WPF is and this library is not C#. Maybe you're using some unofficial fork?

felipedrumond commented 1 year ago

Sorry - this was supposed to be in TiktokenSharp (C#) nuget package.