Issue with nodejs_compat + dynamic require of node:events

[dja]

Which Cloudflare product(s) does this pertain to?

Workers/Other, Wrangler

What version of Wrangler are you using?

2.18.0

What operating system are you using?

Mac

Describe the Bug

Attempting to upload a worker with Wrangler using compatibility_flags = ['nodejs_compat'] or command line --compatibility-flags="nodejs_compat" fails with this error message:

  Uncaught Error: Dynamic require of "node:events" is not supported
    at worker.js:13:11
    at worker.js:13348:28
    at worker.js:13631:3
   [code: 10021]

The package using node:events is this LaunchDarkly SDK: https://github.com/launchdarkly/js-core/tree/main/packages/sdk/cloudflare

[ra-lfaro]

Same exact issue for me as well using same package

[dcrodman]

Having the same issue with trying to use import { Buffer } from "node:buffer"; per the docs:

service core:user:worker: Uncaught Error: Dynamic require of "node:buffer" is not supported
  at core:user:worker:14:11
  at core:user:worker:1749:28
  at core:user:worker:2897:3

This was with Wrangler version v.2.20.0 as well as v3.0.0. Also encountering this error with wrangler dev.

[ra-lfaro]

Anyone have any luck resolving this?

[petebacondarwin]

Right now, the only way to access these compatible libraries is by a static import {Buffer} from "node:buffer"; statement or a dynamic await import("node:Buffer") expression. Not a require() I believe.

[dcrodman]

I'm using import { Buffer } from "node:buffer"; (as opposed to require()) as the docs recommended and am still encountering the issue

[jasnell]

Which workers syntax are y'all using? The older "server worker" syntax (using addEventListener) or the new ESM syntax e.g. export default { async fetch .... } ? If you have a simple repro script handy that doesn't use any specific dependencies that'll be helpful too.

[dcrodman]

@jasnell here's useless but functional barebones example where I see the problem:

worker.ts

import { Buffer } from 'node:buffer';

addEventListener('fetch', (event) => {
    let fetchEvent: FetchEvent = event as FetchEvent;
    fetchEvent.respondWith(handle(fetchEvent));
});

async function handle(event: FetchEvent): Promise<Response> {
    let b = Buffer.from(event.request.body);
    console.log(b);
    return new Response(event.request.body);
}

wrangler.toml

name = "cloudflare-broken-node-demo"
main = "src/worker.ts"
compatibility_date = "2023-07-10"
compatibility_flags = [ "nodejs_compat" ]

package.json

{
  "name": "cloudflare-broken-node-demo",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "deploy": "wrangler deploy",
    "start": "wrangler dev"
  },
  "devDependencies": {
    "@cloudflare/workers-types": "^4.20230419.0",
    "typescript": "^5.0.4",
    "wrangler": "^3.0.0"
  }
}

I haven't deployed this to the actual worker instance, but I see the issue I was seeing before with miniflare locally:

❯ npx wrangler dev
▲ [WARNING] It looks like you have used Wrangler v1's `config` command to login with an API token.

  This is no longer supported in the current version of Wrangler.
  If you wish to authenticate via an API token then please set the `CLOUDFLARE_API_TOKEN`
  environment variable.

 ⛅️ wrangler 3.2.0
------------------
wrangler dev now uses local mode by default, powered by 🔥 Miniflare and 👷 workerd.
To run an edge preview session for your Worker, use wrangler dev --remote
⎔ Starting local server...
service core:user:cloudflare-broken-node-demo: Uncaught Error: Dynamic require of "node:buffer" is not supported
  at core:user:cloudflare-broken-node-demo:8:11
  at core:user:cloudflare-broken-node-demo:256:28
  at core:user:cloudflare-broken-node-demo:266:3
✘ [ERROR] MiniflareCoreError [ERR_RUNTIME_FAILURE]: The Workers runtime failed to start. There is likely additional logging output above.

[irvinebroque]

@dcrodman — looks like this issue is specific to the legacy service worker syntax. I can reproduce it, but only when using the addEventListener syntax.

addEventListener('fetch', (event) => {
    let fetchEvent: FetchEvent = event as FetchEvent;
    fetchEvent.respondWith(handle(fetchEvent));
});

Instead, please use the ESM syntax:

export default {
    async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
        let b = Buffer.from(request.body);
        console.log(b);
        return new Response('Hello World!');
    },
};

I'll make sure to update our docs to be clearer about this, and we'll take a look at our error messages in this case.

For more on ESM syntax, and to migrate from using service workers, refer to: https://developers.cloudflare.com/workers/learning/migrate-to-module-workers/

[dcrodman]

Thank you @irvinebroque , the docs update feels like the most important part here since this can create a potentially difficult situation for people trying to upgrade to Wrangler 3. Migrating is still not necessarily easy and knowing this up front would've been super helpful.

[irvinebroque]

I think my statement above was misleading and not quite right.

Simple reproduction of the issue:

https://github.com/irvinebroque/nodejs_compat_issue/blob/main/src/worker.ts

// This does not work
const { Writable } = require('node:stream');

// This works
// import { Writable } from "node:stream";

const myStream = new Writable();

export default {
    async fetch(request) {
        return new Response("hello");
    },
};

We will take a look at what we can do here.

[mrbbot]

next-on-pages has a nice fix for this issue we could consider applying to all Wrangler users: https://github.com/cloudflare/next-on-pages/blob/7a18efb5cab4d86c8e3e222fc94ea88ac05baffd/packages/next-on-pages/src/buildApplication/processVercelFunctions/build.ts#L86-L112. I think this will only fix the case of require() inside an ES module worker though.

[g-wozniak]

The issue still persists in the latest @sveltejs/adapter-cloudflare :(

[jasnell]

@irvinebroque ... the require(...) method is not present in regular ESM modules so I wouldn't expect that to work. The require(...) method is only injected into common js and node.js compat modules

[irvinebroque]

Ah right, unless the userland code is uploaded as a Node.js module type (https://github.com/cloudflare/workerd/pull/564, only supported in workerd, not by surrounding tooling) or CommonJS module type (only used if manually configured AFAIK) — then you can't use require() directly.

But that because Wrangler runs esbuild, it effectively papers over this — it looks like require() is supported but that's really just bundler magic.

Scaffolding out some better docs in https://github.com/cloudflare/cloudflare-docs/pull/13344

[jasnell]

I'm going to remove the bug label from this issue as this really isn't a bug in the runtime as much as it is just a limitation of the current implementation. I think we can recharacterize this, however, as a possible feature request asking for the module registry to work even with the old service worker syntax.

[noahfraiture]

Does anyone have a solution ?

[vtempest]

I am having this issue trying to run cheerio :(

import * as cheerio from "cheerio";

[xcaptain]

How to solve this? I encountered Error: Dynamic require of "node:buffer" is not supported, I think it's caused by @solana/web3.js package

[vtempest]

You can't. You have to either use a different platform or rewrite the original library's code to not use that part. Hopefully it is not needed. I am not sure why cfw has these limits that cripple most NPM... why they cannot just run Bun.js in Alpine?

[xcaptain]

I fixed my Error: Dynamic require of "node:buffer" is not supported by using a newer version and removed some legacy dependencies. Then I faced Buffer is not defined, I solved this by installing vite-plugin-node-polyfills

I'm using vite + sveltekit + cloudflare pages

[sosnowski]

@xcaptain , can you share an example or POC of your solution? I'm also fighting with Sveltekit and cloudflare pages :/

[vtempest]

Yeah when I do that vite-plugin-node-polyfills and add it to vite config then it gives me even worse error in Bun.js with wrangler 3.57 (newer versions also cause stack error)

`

<--- Last few GCs --->

[8792:000001DF409FF5D0] 254870 ms: Scavenge (reduce) 4069.2 (4142.8) -> 4069.0 (4143.5) MB, 3273.80 / 0.00 ms (average mu = 0.603, current mu = 0.411) allocation failure; [8792:000001DF409FF5D0] 260917 ms: Mark-Compact (reduce) 4070.0 (4143.5) -> 4069.5 (4144.2) MB, 6044.19 / 0.00 ms (average mu = 0.454, current mu = 0.352) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory ----- Native stack trace -----

1: 00007FF6FF7A39EB node::SetCppgcReference+18459 2: 00007FF6FF713238 DSA_meth_get_flags+93160 3: 00007FF70018C771 v8::Isolate::ReportExternalAllocationLimitReached+65
4: 00007FF700175EC8 v8::Function::Experimental_IsNopFunction+1336 5: 00007FF6FFFD7970 v8::Platform::SystemClockTimeMillis+659328 6: 00007FF6FFFD49F8 v8::Platform::SystemClockTimeMillis+647176 7: 00007FF6FFFE9D0A v8::Platform::SystemClockTimeMillis+733978 8: 00007FF6FFFEA587 v8::Platform::SystemClockTimeMillis+736151 9: 00007FF6FFFF31BE v8::Platform::SystemClockTimeMillis+772046 10: 00007FF7000079FA v8::Platform::SystemClockTimeMillis+856074 11: 00007FF70000790D v8::Platform::SystemClockTimeMillis+855837 12: 00007FF6FFDD202F v8::base::Thread::StartSynchronously+710143 13: 00007FF6FFDD5464 v8::base::Thread::StartSynchronously+723508 14: 00007FF6FFDD4EDE v8::base::Thread::StartSynchronously+722094 15: 00007FF6FFDD3E45 v8::base::Thread::StartSynchronously+717845 16: 00007FF6FFDD66C8 v8::base::Thread::StartSynchronously+728216 17: 00007FF6FFDD6497 v8::base::Thread::StartSynchronously+727655 18: 00007FF6FFE1F686 v8::base::Thread::StartSynchronously+1027158 19: 00007FF6FFE1F5BC v8::base::Thread::StartSynchronously+1026956 20: 00007FF6FFCB2C9F v8::CodeEvent::GetFunctionName+89951 21: 00007FF70023E0FE v8::PropertyDescriptor::writable+678094 22: 00007FF70021FE3B v8::PropertyDescriptor::writable+554507 23: 00007FF680B45A5B ⛅️ wrangler 3.57.2 (update availablee `

[xcaptain]

@xcaptain , can you share an example or POC of your solution? I'm also fighting with Sveltekit and cloudflare pages :/

You can check my demo repo at: https://github.com/xcaptain/cf-demo1/blob/main/src/routes/test/%2Bserver.ts#L2

Run:

bun i
bun run build
bunx wrangler pages dev .\.svelte-kit\cloudflare

Then: GET localhost:8788/test

Would see error:

❯❯ cf-demo1 git:(main) 17:34 bunx wrangler pages dev .\.svelte-kit\cloudflare

 ⛅️ wrangler 3.78.8
-------------------

✨ Compiled Worker successfully
Your worker has access to the following bindings:
- Vars:
  - BUN_VERSION: "1.1"
[wrangler:inf] Ready on http://127.0.0.1:8788
⎔ Starting local server...
✨ Parsed 2 valid header rules.
✘ [ERROR] Error: Dynamic require of "node:buffer" is not supported

      at __require (file:///C:/Temp/cf-demo1/.wrangler/tmp/dev-yQAHzs/r1at75qs3df.js:6:9)
      at node_modules/safe-buffer/index.js
  (file:///C:/Temp/cf-demo1/node_modules/safe-buffer/index.js:3:14)
      at __require222 (file:///C:/Temp/cf-demo1/.wrangler/tmp/dev-yQAHzs/r1at75qs3df.js:84:51)
      at
  node_modules/@solana/web3.js/node_modules/bs58/node_modules/base-x/src/index.js
  [as js]
  (file:///C:/Temp/cf-demo1/node_modules/@solana/web3.js/node_modules/bs58/node_modules/base-x/src/index.js:8:15)
      at __require222 (file:///C:/Temp/cf-demo1/.wrangler/tmp/dev-yQAHzs/r1at75qs3df.js:84:51)
      at node_modules/@solana/web3.js/node_modules/bs58/index.js [as js]
  (file:///C:/Temp/cf-demo1/node_modules/@solana/web3.js/node_modules/bs58/index.js:1:13)
      at __require222 (file:///C:/Temp/cf-demo1/.wrangler/tmp/dev-yQAHzs/r1at75qs3df.js:84:51)
      at node_modules/@solana/web3.js/lib/index.browser.esm.js [as js]
  (file:///C:/Temp/cf-demo1/.wrangler/tmp/dev-yQAHzs/r1at75qs3df.js:13306:27)
      at __init (file:///C:/Temp/cf-demo1/.wrangler/tmp/dev-yQAHzs/r1at75qs3df.js:81:56)
      at node_modules/@solana/actions/lib/esm/fetchTransaction.js
  (file:///C:/Temp/cf-demo1/.wrangler/tmp/dev-yQAHzs/r1at75qs3df.js:19805:5)

[wrangler:inf] GET /test 500 Internal Server Error (29ms)

The reason is @solana/actions depends on @solana/web3.js@V1 depends on safe-buffer@v5 while in safe-buffeer/index.js:3

var buffer = require('buffer')

require is not found in cloudflare worker.

My solution is to not use @solana/actions and upgrade @solana/web3.js to v2 so not depends on safe-buffer anymore.

[petebacondarwin]

@xcaptain - the problem we have for your example is that the Svelte Cloudflare adaptor is bundling the generated worker code into a single file using this esbuild call: https://github.com/sveltejs/kit/blob/ed6b5cd4fc0d1d299c838628b73027dd07179281/packages/adapter-cloudflare/index.js#L87-L109

                const result = await esbuild.build({
                    platform: 'browser',
                    // https://github.com/cloudflare/workers-sdk/blob/a12b2786ce745f24475174bcec994ad691e65b0f/packages/wrangler/src/deployment-bundle/bundle.ts#L35-L36
                    conditions: ['workerd', 'worker', 'browser'],
                    sourcemap: 'linked',
                    target: 'es2022',
                    entryPoints: [`${tmp}/_worker.js`],
                    outfile: `${dest}/_worker.js`,
                    allowOverwrite: true,
                    format: 'esm',
                    bundle: true,
                    loader: {
                        '.wasm': 'copy',
                        '.woff': 'copy',
                        '.woff2': 'copy',
                        '.ttf': 'copy',
                        '.eot': 'copy',
                        '.otf': 'copy'
                    },
                    external,
                    alias: Object.fromEntries(compatible_node_modules.map((id) => [id, `node:${id}`])),
                    logLevel: 'silent'
                });

What this does is treat all the node module built in imports as external. As such it doesn't attempt to inline the require() calls - instead converts them into __require() calls. But then it also assumes that require will not be available at runtime as adds a little wrapper at the start of the bundle:

var __require = /* @__PURE__ */ ((x) => typeof require !== "undefined" ? require : typeof Proxy !== "undefined" ? new Proxy(x, {
  get: (a, b) => (typeof require !== "undefined" ? require : a)[b]
}) : x)(function(x) {
  if (typeof require !== "undefined") return require.apply(this, arguments);
  throw Error('Dynamic require of "' + x + '" is not supported');
});

This is where the runtime exceptions appear.

[petebacondarwin]

The good news is that as of 2024-09-23 (compat date), we have a new version of nodejs_compat and we also now have Workers Assets that allows us to do the bundling inside Wrangler.

The output from the vite build includes the two directories we really need, which we can configure in wrangler.toml:

name = "cf-demo1"
compatibility_date = "2024-10-04"
compatibility_flags = ["nodejs_compat"]
assets = { directory = ".svelte-kit/output/client" }
main = ".svelte-kit/cloudflare-tmp/_worker.js"

[vars]
BUN_VERSION = "1.1"

Now running wrangler dev will allow us to hit localhost:8787/test without any errors.

[JJK801]

Hi, i add the same problem with Cloudflare Pages and sveltkit adapter, here is a small patch that seems to do the trick:

diff --git a/index.js b/index.js
index 0fe55898e5697fc6a061299780e163ca4553cb05..0bd37d9fda86340ac77585e1334707b2cf0c5c48 100644
--- a/index.js
+++ b/index.js
@@ -81,7 +81,7 @@ export default function (options = {}) {
                }
            });

-           const external = ['cloudflare:*', ...compatible_node_modules.map((id) => `node:${id}`)];
+           const external = ['cloudflare:*', /*...compatible_node_modules.map((id) => `node:${id}`)*/];

            try {
                const result = await esbuild.build({
@@ -104,7 +104,7 @@ export default function (options = {}) {
                        '.otf': 'copy'
                    },
                    external,
-                   alias: Object.fromEntries(compatible_node_modules.map((id) => [id, `node:${id}`])),
+                   //alias: Object.fromEntries(compatible_node_modules.map((id) => [id, `node:${id}`])),
                    logLevel: 'silent'
                });