search

latitudegames / GPT-3-Encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3
MIT License
716 stars 196 forks source link

"TextEncoder is not defined" error when running tests in Node 12.22.0 #15

Open sshadmand opened 1 year ago

sshadmand commented 1 year ago

Package installed

"dependencies": {
    ...
    "gpt-3-encoder": "^1.1.3",
    ...
}

When ran 

    ReferenceError: TextEncoder is not defined 
      at Object.<anonymous> (node_modules/gpt-3-encoder/Encoder.js:21:21)

Which references: https://github.com/latitudegames/GPT-3-Encoder/blob/master/Encoder.js#L21

Suggestion: Potentially allow pass in for TextEncoder or change to NodeJS reco:

var util = require('util');
let encoder = new util.TextEncoder();
let uint8Array = encoder.encode("Hello");
console.log(uint8Array);
sshadmand commented 1 year ago

NodeJS compatible fork PR here: https://github.com/latitudegames/GPT-3-Encoder/pull/16