vercel / next.js

The React Framework
https://nextjs.org
MIT License
125.02k stars 26.7k forks source link

using lodash-es & pnpm & turberepo(monorepo), Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime #51401

Open shinyuna opened 1 year ago

shinyuna commented 1 year ago

Verify canary release

Provide environment information

Operating System:
      Platform: darwin
      Arch: arm64
      Version: Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000
    Binaries:
      Node: 18.15.0
      npm: 9.5.0
      Yarn: 1.22.19
      pnpm: 7.32.4
    Relevant packages:
      next: 13.4.6
      eslint-config-next: 13.4.3
      react: 18.2.0
      react-dom: 18.2.0
      typescript: 4.8.4

Which area(s) of Next.js are affected? (leave empty if unsure)

Middleware / Edge (API routes, runtime)

Link to the code that reproduces this issue or a replay of the bug

https://github.com/inflearn/inflearn-ucc-frontend/pull/14

To Reproduce

In the middleware.ts file, we are importing a file that is using lodash-es.

If you run the next build, you will get an error and the build will fail.

I set the config as suggested in the documentation, but the result is the same.

The documentation even says to set the runtime to edge, but when you run the actual build, you get this error: Error: Page /src/middleware provided runtime 'edge', the edge runtime for rendering is currently experimental. Use runtime 'experimental-edge' instead. You'll find an error like this

Option values I set

export const config = {
  runtime: "experimental-edge",
  unstable_allowDynamic: [
    "/src/utils/Fetcher.ts",
    "/src/@types/schema.ts",
    "*/**/node_modules/lodash-es/**",
  ],
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

Describe the Bug

build will fail.

Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime Used by default Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation

Expected Behavior

You'll likely encounter an error message like this

../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/_root.js
profile:build: Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime 
profile:build: Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation
profile:build: 
profile:build: Import trace for requested module:
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/_root.js
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/isBuffer.js
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/lodash.js
profile:build: ./src/utils/Fetcher.ts
profile:build: ./src/@types/schema.ts
profile:build: 
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/template.js
profile:build: Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime 
profile:build: Used by default
profile:build: Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation
profile:build: 
profile:build: Import trace for requested module:
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/template.js
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/lodash.js
profile:build: ./src/utils/Fetcher.ts
profile:build: ./src/@types/schema.ts

Which browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

shuding commented 1 year ago

As the error message says

Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime
Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation

This is expected as https://unpkg.com/lodash-es@4.17.21/_root.js has a Function('return this')().

jer-k commented 1 year ago

@shuding Can this be-reopened?

I'm running into this same issue and we run our Next.js project in a Docker container in Kubernetes so there is 0% chance the code will run on the Edge Runtime. The code executes perfectly fine in development mode.

My understanding, and what @shinyuna demonstrated in the reproduction, is that unstable_allowDynamic is supposed to allow us to tell Next.js that we aren't worried about the Edge runtime and we should be allowed to have Dynamic Code Evaluation. However, after adding the path to unstable_allowDynamic we are still presented with the error and unable to create a production build. I thought the point of unstable_allowDynamic was to allow the build to work, please let me know if this is not the correct thinking.

jer-k commented 1 year ago

I did some debugging and re-created the issue in this repository https://github.com/jer-k/middleware-debug

On the main branch, everything works as expected

On the yarnpnp branch, I cannot get the build to work correctly

middleware-debug % git checkout 7fe1e2d337c84e644aa5d151624f51e3c5d9f9c9
middleware-debug % yarn build
- info Linting and checking validity of types  
- warn You are using an experimental edge runtime, the API might change.
Failed to compile.

./.yarn/cache/lodash-npm-4.17.21-6382451519-eb835a2e51.zip/node_modules/lodash/lodash.js
Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime 
Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation

Import trace for requested module:
./.yarn/cache/lodash-npm-4.17.21-6382451519-eb835a2e51.zip/node_modules/lodash/lodash.js

> Build failed because of webpack errors
- info Creating an optimized production build .
middleware-debug % git checkout f66ca0febace99c8e34e58f4b45da3ed1e21b4c5
middleware-debug % yarn build
- info Linting and checking validity of types  
- warn You are using an experimental edge runtime, the API might change.
Failed to compile.

./.yarn/cache/lodash-npm-4.17.21-6382451519-eb835a2e51.zip/node_modules/lodash/lodash.js
Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime 
Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation

Import trace for requested module:
./.yarn/cache/lodash-npm-4.17.21-6382451519-eb835a2e51.zip/node_modules/lodash/lodash.js

> Build failed because of webpack errors
- info Creating an optimized production build .

This leads me to believe the issue lies in the isMatch function here https://github.com/vercel/next.js/pull/39539/files#diff-f85f0611a4f531f0d0eb59c9818aff5717e1b1c4523a8ba55c37b263d5a492a4R258

I'm not savvy enough to hook up my repository to a local version of next.js so I could see what is being passed in and debug further, but hopefully this reproduction is enough to get some help here.

Tagging @feugy as it looks like this feature was added via https://github.com/vercel/next.js/pull/39539

Thank you

jer-k commented 1 year ago

Looks like I was able to resolve this. After adding the telemetry output I was able to see this

[telemetry] {
  "eventName": "NEXT_EDGE_ALLOW_DYNAMIC_USED",
  "payload": {
    "file": "private-next-root-dir/src/middleware.ts",
    "config": {
      "unstable_allowDynamicGlobs": [
        "**/.yarn/cache/**/node_modules/lodash/*.js"
      ]
    },
    "fileWithDynamicCode": "/.yarn/cache/lodash-npm-4.17.21-6382451519-eb835a2e51.zip/node_modules/lodash/lodash.js"
  }
}

The matcher from the glob is expecting to match against "/.yarn/cache/lodash-npm-4.17.21-6382451519-eb835a2e51.zip/node_modules/lodash/lodash.js" which has just a leading /. This is different from the logged message which had a leading ./

The fixed absolute url can be seen here https://github.com/jer-k/middleware-debug/commit/20a696e05787b301de5a1c3f1821dcda1cc290dc

Then I changed from an absolute path to a glob path. unstable_allowDynamic: ["**/.yarn/cache/**/node_modules/lodash/*.js"]. It appears that the ** matcher doesn't match on hidden directories, such as .yarn/ thus why the original **/node_modules/lodash/lodash.js pattern did not allow the build to successfully finish. The change can be seen here https://github.com/jer-k/middleware-debug/commit/664a96b50e32f298734534f3541e86f5ea3b3cf2

The solution might be a adjusting the logging to match what is passed into fileWithDynamicCode?

grantmagdanz commented 1 year ago

@jer-k how did you get that telemetry output? I'm running into the same issue but can't figure out what is wrong. I'm using pnpm and not yarnpnp. I tried with NEXT_TELEMETRY_DEBUG=1 but didn't see your output.

jer-k commented 1 year ago

@grantmagdanz You have to include the config with unstable_allowDynamic in your middleware file to get the telemetry to print out. As long as there is anything in there, it'll print.

export const config = {
    runtime: "experimental-edge",
    unstable_allowDynamic: ["**/node_modules/*.js"],
};

should suffice to get it printing. It only prints on the build command, so yarn build for me. Hope that helps!

shinyuna commented 1 year ago

I still get the same error.

The config value I set:

export const config = {
  runtime: "experimental-edge",
  unstable_allowDynamic: [ [
    "/node_modules/lodash-es/**",
    "/.pnpm/**/node_modules/lodash-es/*.js",
  ],
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

error log:

d: Failed to compile.
profile:build: 
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/_root.js
profile:build: Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime 
profile:build: Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation
profile:build: 
profile:build: Import trace for requested module:
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/_root.js
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/isBuffer.js
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/lodash.js
profile:build: ./src/utils/Fetcher.ts
profile:build: ./src/@types/schema.ts
profile:build: 
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/template.js
profile:build: Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime 
profile:build: Used by default
profile:build: Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation
profile:build: 
profile:build: Import trace for requested module:
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/template.js
profile:build: ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/lodash.js
profile:build: ./src/utils/Fetcher.ts
profile:build: ./src/@types/schema.ts
profile:build: 
jer-k commented 1 year ago

@shinyuna Try "**/.pnpm/**/node_modules/lodash-es/*.js" you need to capture the ../../node_modules at the start of ../../node_modules/.pnpm/lodash-es@4.17.21/node_modules/lodash-es/_root.js which ** will match

grantmagdanz commented 1 year ago

In case anyone else runs into this and is using Sentry, withSentryConfig breaks unstable_allowDynamic. Everything else in the above is also true but I ran into issues because of Sentry. Filed an issue with Sentry: https://github.com/getsentry/sentry-javascript/issues/8386

shuding commented 1 year ago

I think on the Next.js side, we will try to compile Function('return this')() into non-dynamic code (globalThis) in the Edge Runtime. In the meantime, this seems like also a bug in unstable_allowDynamic that needs to be addressed.

I'm running into this same issue and we run our Next.js project in a Docker container in Kubernetes so there is 0% chance the code will run on the Edge Runtime.

Edge Runtime is not an infra, and it will be the runtime for Next.js middleware, and pages with export const runtime = 'edge', no matter where you host the Next.js project.

jer-k commented 1 year ago

Edge Runtime is not an infra, and it will be the runtime for Next.js middleware, and pages with export const runtime = 'edge', no matter where you host the Next.js project.

I see. I was misunderstanding what edge was and thinking it meant https://vercel.com/features/edge-functions. Thank you for the clarification

elartix commented 4 months ago

Hi folks, caught regression issue with unstable_allowDynamic feature

"next": "14.3.0-canary.16" - failed too "next": "14.2.2" - failed "next": "14.2.1" - failed "next": "14.2.0" - failed "next": "14.1.4" - ok "next": "14.1.3" - ok

// middleware.ts

// only applies this middleware to files in the app directory
export const config = {
  runtime: 'experimental-edge',  // 'nodejs' (not allowed 14.x) | 'edge' | 'experimental-edge'
  unstable_allowDynamic: [
    // '**/node_modules/*.js'
    // '**/node_modules/lodash/lodash.js'
    '**/node_modules/lodash/**',
  ],
  matcher: '/((?!api|static|.*\\..*|_next|assets|favicon.ico|sw.js|.*\\.(?:svg|png|jpg|jpeg|gif|webp)$).*)'
};
// output

./node_modules\lodash\lodash.js
Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime
Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation

Import trace for requested module:
  ./node_modules\lodash\lodash.js
  ./src\services\auth\auth.ts
  ./src\middleware.ts

> Build failed because of webpack errors
Zhaph commented 4 months ago

We've just had the same issue referencing Bloom-Filter which references lodash and by extension reflect-metadata after we upgraded from Next 13.4.1 to Next 14.2.0

  unstable_allowDynamic: [
    '/node_modules/lodash/**',
    '/node_modules/reflect-metadata/**',
  ]

Worked fine before the upgrade to allow our builds to continue and the references to bloom-filter to work as expected, however after upgrading we're unable to build on Windows - interestingly it still worked as expected on Ubuntu (therefore not an issue for our actual deployments):

./node_modules\lodash\lodash.js
Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime
Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation

Import trace for requested module:
  ./node_modules\lodash\lodash.js
  ./node_modules\bloom-filters\dist\sketch\topk.js
  ./node_modules\bloom-filters\dist\api.js
  ./src\lib\middleware\plugins\bulkRedirects.ts
  ./src\temp\middleware-plugins.ts
  ./src\lib\middleware\index.ts
  ./src\middleware.ts

./node_modules\reflect-metadata\Reflect.js
Dynamic Code Evaluation (e. g. 'eval', 'new Function', 'WebAssembly.compile') not allowed in Edge Runtime
Learn More: https://nextjs.org/docs/messages/edge-dynamic-code-evaluation

Import trace for requested module:
  ./node_modules\reflect-metadata\Reflect.js
  ./node_modules\bloom-filters\dist\exportable.js
  ./node_modules\bloom-filters\dist\bloom\bloom-filter.js
  ./node_modules\bloom-filters\dist\api.js
  ./src\lib\middleware\plugins\bulkRedirects.ts
  ./src\temp\middleware-plugins.ts
  ./src\lib\middleware\index.ts
  ./src\middleware.ts

I was able to get it to work on Windows as well by explicitly referencing the problematic files:

  unstable_allowDynamic: [
    '/node_modules/lodash/**',
    '/node_modules/reflect-metadata/**',
    '\\node_modules\\lodash\\lodash.js',
    '\\node_modules\\reflect-metadata\\Reflect.js',
  ]

Telemetry output if it helps:

  "eventName": "NEXT_EDGE_ALLOW_DYNAMIC_USED",
  "payload": {
    "file": "private-next-root-dir/src/middleware.ts",
    "config": {
      "matchers": [
        {
          "regexp": "^(?:\\/(_next\\/data\\/[^/]{1,}))?(?:\\/((?!_next\\/)[^/.]{1,}))(|\\.json|\\/?index|\\/?index\\.json)?[\\/#\\?]?$",
          "originalSource": "/"
        },
        {
          "regexp": "^(?:\\/(_next\\/data\\/[^/]{1,}))?(?:\\/((?!_next\\/)[^/.]{1,}))(?:\\/((?!api\\/|_next\\/|healthz|people\\/|insights\\/|sitecore\\/api\\/|-\\/|favicon.ico|sc_logo.svg).*))(.json)?[\\/#\\?]?$",
          "originalSource": "/((?!api/|_next/|healthz|people/|insights/|sitecore/api/|-/|favicon.ico|sc_logo.svg).*)"
        }
      ],
      "unstable_allowDynamicGlobs": [
        "/node_modules/lodash/**",
        "/node_modules/reflect-metadata/**",
        "\\node_modules\\lodash\\*.js",
        "\\node_modules\\reflect-metadata\\*.js"
      ]
    },
    "fileWithDynamicCode": "\\node_modules\\lodash\\lodash.js"
  }
}
[telemetry] {
  "eventName": "NEXT_EDGE_ALLOW_DYNAMIC_USED",
  "payload": {
    "file": "private-next-root-dir/src/middleware.ts",
    "config": {
      "matchers": [
        {
          "regexp": "^(?:\\/(_next\\/data\\/[^/]{1,}))?(?:\\/((?!_next\\/)[^/.]{1,}))(|\\.json|\\/?index|\\/?index\\.json)?[\\/#\\?]?$",
          "originalSource": "/"
        },
        {
          "regexp": "^(?:\\/(_next\\/data\\/[^/]{1,}))?(?:\\/((?!_next\\/)[^/.]{1,}))(?:\\/((?!api\\/|_next\\/|healthz|people\\/|insights\\/|sitecore\\/api\\/|-\\/|favicon.ico|sc_logo.svg).*))(.json)?[\\/#\\?]?$",
          "originalSource": "/((?!api/|_next/|healthz|people/|insights/|sitecore/api/|-/|favicon.ico|sc_logo.svg).*)"
        }
      ],
      "unstable_allowDynamicGlobs": [
        "/node_modules/lodash/**",
        "/node_modules/reflect-metadata/**",
        "\\node_modules\\lodash\\*.js",
        "\\node_modules\\reflect-metadata\\*.js"
      ]
    },
    "fileWithDynamicCode": "\\node_modules\\reflect-metadata\\Reflect.js"
  }
}
WhiteMinds commented 3 weeks ago

I also encountered this issue when upgrading next.js from 14.1.0 to 14.2.6.

This is a PR attempting to fix it: https://github.com/vercel/next.js/pull/69402.