search

ceifa / tiktoken-node

OpenAI's tiktoken but with node bindings
102 stars 10 forks source link

Not working in deno deploy #3

Open deepak-coding-art opened 1 year ago

deepak-coding-art commented 1 year ago

I am trying to load it in deno with the help of esm.sh but it is giving error:

error: Uncaught Error: createRequire only supports 'file://' URLs for the 'filename' parameter. Received 'https://esm.sh/v111/tiktoken-node@0.0.4/deno/tiktoken-node.js'
          throw new Error(
                ^
    at createRequire (https://deno.land/std@0.177.0/node/module.ts:690:17)
    at https://esm.sh/v111/tiktoken-node@0.0.4/deno/tiktoken-node.js:2:185

import statement

import tiktoken from "https://esm.sh/tiktoken-node@0.0.4";
ceifa commented 1 year ago

Looks like the createRequire function have a different behavior on deno. Anyway using compat mode fixes it:

import tiktoken from "npm:tiktoken-node";
deepak-coding-art commented 1 year ago

I am using the supabase edge functions and the npm packages are not available in that

Did this package support the cl100k_base encoding for the gpt-turbo-3.5 model

waptik commented 1 year ago

I am trying to load it in deno with the help of esm.sh but it is giving error:

error: Uncaught Error: createRequire only supports 'file://' URLs for the 'filename' parameter. Received 'https://esm.sh/v111/tiktoken-node@0.0.4/deno/tiktoken-node.js'
          throw new Error(
                ^
    at createRequire (https://deno.land/std@0.177.0/node/module.ts:690:17)
    at https://esm.sh/v111/tiktoken-node@0.0.4/deno/tiktoken-node.js:2:185

import statement

import tiktoken from "https://esm.sh/tiktoken-node@0.0.4";

Did you try using skypack cdn instead of esm.sh? Sometimes using npm packages with skypack.dev works where esm fails

liamcharmer commented 1 year ago

I am using the supabase edge functions and the npm packages are not available in that

Did this package support the cl100k_base encoding for the gpt-turbo-3.5 model

Did you get this working in the end?