search

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240 stars 47 forks source link

Add gpt-4-1106-preview model #49

Closed JosephTLyons closed 10 months ago

JosephTLyons commented 10 months ago

I'm not entirely sure if anything else here is needed. I opened a PR to the tiktoken submodule here - assuming that might need to be merged in first before this one could be considered?

zurawiki commented 10 months ago

Thanks for the PR! Linking to the docs here. This tiktoken-rs module doesn't depend on the python code. To get around that, we rewrote some of the wrapping python logic in rust for consistency.

JosephTLyons commented 10 months ago

Awesome - Thanks for the info and for accepting the PR!