Open gch116620 opened 8 months ago
When using gpt tokens to calculate tokens, I found that if the content contains values similar to<| endoftext |>, the calculation will report an error. How about use a filtering function like text=text. replace (/<|\w{0,} |>/g, "").
When using gpt tokens to calculate tokens, I found that if the content contains values similar to<| endoftext |>, the calculation will report an error. How about use a filtering function like text=text. replace (/<|\w{0,} |>/g, "").