Non-English Characters Can Consume More Tokens - Githubissues

TomFrankly / pipedream-notion-voice-notes

Take notes with your voice and send them to Notion

103 stars 55 forks source link

Non-English Characters Can Consume More Tokens #38

Closed TomFrankly closed 11 months ago

TomFrankly commented 11 months ago

https://community.openai.com/t/all-languages-are-not-created-tokenized-equal/216407/6