Review GPToken

[syonfox]

Making a new branch for development GPToken ... master seems to work for vanilla and browser and be stable

[Shackless]

I'm trying to use it in SvelteKit/TS and get this error:

import { countTokens as countTokensExt } from 'gptoken';

function whatever() {
   const num_tokens += countTokensExt("Oh Hello");
}

Error: Buffer is not defined:

[syonfox]

Hmm does svelte kit not have Buffer is node the backend I was worried changing that dependency might break some environment... Perhaps we need to include a pollyfill or in svelte you need to install the buffer dependency... You could try using the browser version too.

On Wed., Mar. 15, 2023, 07:28 Shackless, @.***> wrote:

I'm trying to use it in SvelteKit/TS and get this error:

import { countTokens as countTokensExt } from 'gptoken'; function whatever() { const num_tokens += countTokensExt("Oh Hello");}

Error: Buffer is not defined:

[image: image] https://user-images.githubusercontent.com/306996/225281639-35e08dfe-34df-42a9-abfd-8d05ff6b08d2.png

— Reply to this email directly, view it on GitHub https://github.com/syonfox/GPT-3-Encoder/issues/3#issuecomment-1469746785, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHJVDO7KHCMCPGLNRPNUKTW4GKUNANCNFSM6AAAAAAVVY4WOI . You are receiving this because you authored the thread.Message ID: @.***>

[Shackless]

Oh yeah, I'm trying to run it in the browser (client-side). I installed the Buffer browser version but it sets window.buffer instead of window.Buffer (capital B). Would be great if you could get rid of this dependency and make your package standalone and runnable without Node.

[syonfox]

Have you tried including the bundled source in you client side? browser.js not index.js

I belive browserify does generate polyfills for any node thing used. see #4

[z0d14c]

I also cannot use because of Buffer is not defined

[syonfox]

I also cannot use it because of Buffer is not defined

Can you give me any more information? Are you sure its not the same issue where you are including the node code in the browser instead of browser.js

[syonfox]

https://github.com/feross/buffer

With browserify, simply require('buffer') or use the Buffer global and you will get this module.

I am using Browserify so you should get a version with polyfill