Allow Browser Usage, countTokens convenience method

latitudegames / GPT-3-Encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3

MIT License

716 stars 196 forks source link

Open wesmcouch opened 1 year ago

wesmcouch commented 1 year ago

Remove fs and path usage to allow usage of the encoder and decoder both inside a browser and in nodejs
Add a convenience method named countTokens that counts the number of tokens in a given string
Update the example and tests to include the new method countTokens

nem035 commented 1 year ago

FYI - I published a version to use (works in browser/node but no countTokens) until the maintainers merged: https://www.npmjs.com/package/@nem035/gpt-3-encoder

arcticfly commented 1 year ago

@nem035 looks like you can merge?

astoilkov commented 1 year ago

This PR also makes it work in Electron (not just the browser). Vite also has a problem with the code in master because it uses __dirname.