search

syonfox / gptoken

Javascript BPE Encoder Decoder for GPT-2 / GPT-3
MIT License
24 stars 1 forks source link

Browser and Vanilla JS #4

Open jordymeow opened 1 year ago

jordymeow commented 1 year ago

Hi,

This seems nice but I am not sure; are you currently converting this for vanilla JS, to run this into the browser? If yes, where, which branch? Can I try it already, or it's not ready? Thanks :)

syonfox commented 1 year ago

I believe 

npm install gptoken

<script type="application/javascript" src="./node_modules/gptoken/browser.js"></script>

<script> 
console.log(gptoken.countTokens("hello world"))
</script>

the build is npm run build

and it outputs browser.js

this should be included in the npm package

syonfox commented 1 year ago 
        const tokens = gpt3encoder.encode('Hello, world!');
        const text = gpt3encoder.decode(tokens);
        const tokenCount = gpt3encoder.countTokens('Hello, world!'); 
    It used to be gpt3encoder but I will renam the browser export to gptoken and rerelease now npm version 0.0.4