Cainier / gpt-tokens

Calculate the token consumption and amount of openai gpt message
MIT License
106 stars 13 forks source link

Throw an error when content contains <|endoftext|> #43

Open gch116620 opened 8 months ago

gch116620 commented 8 months ago

When using gpt tokens to calculate tokens, I found that if the content contains values similar to<| endoftext |>, the calculation will report an error. How about use a filtering function like text=text. replace (/<|\w{0,} |>/g, "").