Throws Exception in AWS Lambda

[monostere0]

Couldn't find anything documented, that's why I'm opening an issue here

INIT_START Runtime Version: nodejs:18.v6    Runtime Version ARN: arn:aws:lambda:eu-central-1::runtime:a19cefba18c0aa228bfe71e81d95cc5aab18e4570e899b3cfd9f759f022c7f8c
--

ERROR   Uncaught Exception  
{
    "errorType": "Error",
    "errorMessage": "Missing tiktoken_bg.wasm",
    "stack": [
        "Error: Missing tiktoken_bg.wasm",
        "    at ../backend/node_modules/@dqbd/tiktoken/tiktoken.cjs (/var/task/index.js:529075:13)",
        "    at __require (/var/task/index.js:41:44)",
        "    at Object.<anonymous> (/var/task/index.js:532514:34)",
        "    at Module._compile (node:internal/modules/cjs/loader:1254:14)",
        "    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)",
        "    at Module.load (node:internal/modules/cjs/loader:1117:32)",
        "    at Module._load (node:internal/modules/cjs/loader:958:12)",
        "    at Module.require (node:internal/modules/cjs/loader:1141:19)",
        "    at require (node:internal/modules/cjs/helpers:110:18)",
        "    at _tryRequireFile (file:///var/runtime/index.mjs:912:37)"
    ]
}

I'm thinking I'm maybe missing a lambda layer but I could only find one for Python. Any pointers would be greatly appreciated, thanks!

[dqbd]

Hello @monostere0!

The error does indicate that the WASM file is not bundled properly. If possible, a reproducible example would be great!

[monostere0]

Hi @dqbd, thanks for your reply! Indeed, after drilling more into this issue, it seems like CDK doesn't allow for plugins configuration in its esbuild settings for lambdas. So basically there's no way of doing this with CDK out of the box, however I did find this package which promises to be a workaround. Didn't get the chance to use it yet though, I'll report back once I do.

[bmul]

I had the same problem. One solution that worked was to copy the webassembly into the lambda build directory as part of the deployment. Another was to use the pure JS implementation js-tiktoken

[monostere0]

Thanks for the solutions @bmul!

I haven't managed to have it running, so I've resorted to using the openai npm package, which does what I need. Was getting pretty close with having the wasm package embedded into the bundle itself but I kept running into other issues so I swapped it. I guess this issue can be closed, thanks.