search

dqbd / tiktoken

JS port and JS/WASM bindings for openai/tiktoken
MIT License
649 stars 49 forks source link

request for npm gpt-3.5-turbo support #17

Closed temberature closed 1 year ago

temberature commented 1 year ago

I found that there is already a gpt-3.5-turbo branch, but it is not released to https://www.npmjs.com, I would like to ask about when it will be released?

BTW. Thanks for your great work.

dqbd commented 1 year ago

Still working to brush out the breaking changes due to bundling stuff, but in the meantime, either:

  1. Use get_encoding("cl100k_base"), which should correspond to gpt-3.5-turbo
  2. Check out the 1.0.0-alpha.3 version, which should add support for encoding_for_model("gpt-3.5-turbo")

ETA for latest version is between 2 and 3 days 😄

temberature commented 1 year ago

I attempted using the first cl100k_base, and it appears to yield the same outcome as reported in https://github.com/openai/tiktoken/issues/45.

Thank you for your kind reply~

dqbd commented 1 year ago

Version 1.0.0 has been published, which should add the support for gpt-3.5-turbo, gpt-4 and gpt-4-32k models. LMK if the issue presists.