search

dqbd / tiktoken

JS port and JS/WASM bindings for openai/tiktoken
MIT License
705 stars 53 forks source link

Plans for `gpt-3.5-turbo-16k`? #59

Closed khill-fbmc closed 1 year ago

khill-fbmc commented 1 year ago

I would try to submit a PR, but have no idea how the library works.

vlrevolution commented 1 year ago

Would like to see this implemented as well and the library kept up to date with the changes in api

nikwen commented 1 year ago

I'd assume the tokenizer is the same as for gpt-3.5-turbo.

vlrevolution commented 1 year ago

Yeah actually it is

On Fri, 4 Aug 2023, 21:04 Niklas Wenzel, @.***> wrote:

I'd assume the tokenizer is the same as for gpt-3.5-turbo.

— Reply to this email directly, view it on GitHub https://github.com/dqbd/tiktoken/issues/59#issuecomment-1665991633, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQC2ZKUWRUAIMMIKR2ZARSDXTU2UDANCNFSM6AAAAAAZ5V3GPY . You are receiving this because you commented.Message ID: @.***>

dqbd commented 1 year ago

Hello! Both the js-tiktoken@1.0.7 version and tiktoken@1.0.10 version should have support for gpt-3.5-turbo-16k already! Let me know, if it's not working properly.

merlindru commented 1 year ago

For anyone coming here, please make sure to use the tiktoken library, not @dqbd/tiktoken, which isn't being updated anymore

Could we put up a warning on the NPM page of @dqbd/tiktoken?