error:fetch is not undefined

[boygoboy]

const { Tiktoken } = require("tiktoken/lite"); const { load } = require("tiktoken/load"); const registry = require("tiktoken/registry.json"); const models = require("tiktoken/model_to_encoding.json");

async function main() { const model = await load(registry[models["gpt-3.5-turbo"]]); const encoder = new Tiktoken( model.bpe_ranks, model.special_tokens, model.pat_str ); const tokens = encoder.encode("hello world"); encoder.free(); }

main();

Run the above code in express and report fetch is not undefined. Is this package not available in the nodjs 14 environment?

[dqbd]

Hello @boygoboy!

Yes, the package uses fetch when using tiktoken/load, haven't added a hard engine dependency to allow package consumption when loading the full package.

Consider adding a polyfill such as this: https://github.com/node-fetch/node-fetch#providing-global-access

[boygoboy]

你好@boygoboy！

是的，包fetch在使用时使用tiktoken/load，没有添加硬engine依赖以允许在加载完整包时使用包。

考虑添加这样的 polyfill：https ://github.com/node-fetch/node-fetch#providing-global-access

thanks i'll try it

[dqbd]

Closing the issue for now! Feel free to reopen the issue if you have additional questions :)