Closed wongjingping closed 2 months ago
Update openai tokenizer to use "cl100k_base" encoding since all of their models use it now.
Update openai tokenizer to use "cl100k_base" encoding since all of their models use it now.