When running 1.0.7 there is a Missing tiktoken_bg.wasm error on AWS Lambda typescript

[haibh-wanosoft]

I'm using the @dqbd/tiktoken library to count tokens in a text passage. When I deploy it on a Node.js 18 AWS EC2 instance, it works fine. However, when I deploy the library on an AWS Lambda function with TypeScript, it throws an error.

Code:

import { encoding_for_model } from "@dqbd/tiktoken"; 
const encodingForModelGPT4 = encoding_for_model("gpt-4"); 
encodingForModelGPT4.encode(value);
encodingForModelGPT4.free();

Error

{
  "error": {
    "Error": "Error",
    "Cause": "{\"errorType\":\"Error\",\"errorMessage\":\"Missing tiktoken_bg.wasm\",\"trace\":[\"Error: Missing tiktoken_bg.wasm\",\"    at asset-input/node_modules/@dqbd/tiktoken/tiktoken.cjs (/var/task/index.js:458:13)\",\"    at __require (/var/task/index.js:9:50)\",\"    at Object.<anonymous> (/var/task/index.js:474:31)\",\"    at Module._compile (node:internal/modules/cjs/loader:1254:14)\",\"    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)\",\"    at Module.load (node:internal/modules/cjs/loader:1117:32)\",\"    at Module._load (node:internal/modules/cjs/loader:958:12)\",\"    at Module.require (node:internal/modules/cjs/loader:1141:19)\",\"    at require (node:internal/modules/cjs/helpers:110:18)\",\"    at _tryRequireFile (file:///var/runtime/index.mjs:912:37)\"]}"
  }
}

[haibh-wanosoft]

@dqbd Can you help me? thanks you.

[beatrizdemiguelperez]

we have the same issue with lambda and node 18, Can you help please?

[dqbd]

Hello! Sorry for the delay. The code is highly dependent on the way how your lambda function is deployed. You might want to initialise the WASM and the library manually, similarly as seen in https://github.com/dqbd/tiktoken#vercel-edge-runtime

Alternatively, consider using js-tiktoken instead, which is a pure JS port of @dqbd/tiktoken without WASM dependencies.