Add a way to count tokens without encoding the whole text; improve performance; fix bugs

latitudegames / GPT-3-Encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3

MIT License

716 stars 196 forks source link

Open niieani opened 1 year ago

niieani commented 1 year ago

adds highly performant isWithinTokenLimit to count tokens without encoding the whole text
improve overall performance by removing transitive arrays
include precomputed bpeRanks
add type-checking
fix a few minor bugs (thanks to type-checking)
add generator versions of both decoder and encoder
adds prettier and prettifies the codebase

To test, see fork published as gpt-tokenizer.