search

botisan-ai / gpt3-tokenizer

Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
MIT License
171 stars 19 forks source link

FIX: Prevent "hasOwnProperty" tokens from breaking encoding #14

Closed kopertop closed 1 year ago

kopertop commented 1 year ago

Fixes #13