Next.js fails to find/load native node modules compiled as a separate package in a (pnpm) workspace

daanboer commented 1 month ago

Link to the code that reproduces this issue

https://github.com/daanboer/next-load-native-module

To Reproduce

Install dependencies.

pnpm i

Build the rust-lib package (needs a working rust installation).

pnpm -C packages/rust-lib build

Try to build the web-app package.

pnpm -C packages/web-app build

Current vs. Expected behavior

I expect the greet function from the rust-lib package to be imported and called. Instead the web-app build process throws the following error: [cause]: Error: Cannot find module '../rust-lib/dist/index.node'.

Full build logs

``` > web-app@0.1.0 build /home/daan/git/next-load-native-module/packages/web-app > next build ▲ Next.js 15.0.0 Creating an optimized production build ... ✓ Compiled successfully ✓ Linting and checking validity of types Collecting page data .Error: Failed to collect configuration for / at /home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/build/utils.js:1130:23 at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async Span.traceAsyncFn (/home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/trace/trace.js:157:20) { [cause]: Error: Cannot find module '../rust-lib/dist/index.node' Require stack: - /home/daan/git/next-load-native-module/packages/web-app/.next/server/app/page.js - /home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/server/require.js - /home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/server/load-components.js - /home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/build/utils.js - /home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/build/worker.js - /home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/compiled/jest-worker/processChild.js at Module._resolveFilename (node:internal/modules/cjs/loader:1248:15) at /home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/server/require-hook.js:55:36 at Module._load (node:internal/modules/cjs/loader:1074:27) at TracingChannel.traceSync (node:diagnostics_channel:315:14) at wrapModuleLoad (node:internal/modules/cjs/loader:217:24) at Module.require (node:internal/modules/cjs/loader:1339:12) at mod.require (/home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/server/require-hook.js:65:28) at require (node:internal/modules/helpers:135:16) at 7215 (/home/daan/git/next-load-native-module/packages/web-app/.next/server/app/page.js:1:31238) at t (/home/daan/git/next-load-native-module/packages/web-app/.next/server/webpack-runtime.js:1:127) { code: 'MODULE_NOT_FOUND', requireStack: [ '/home/daan/git/next-load-native-module/packages/web-app/.next/server/app/page.js', '/home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/server/require.js', '/home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/server/load-components.js', '/home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/build/utils.js', '/home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/build/worker.js', '/home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/compiled/jest-worker/processChild.js' ] } } > Build error occurred Error: Failed to collect page data for / at /home/daan/git/next-load-native-module/node_modules/.pnpm/next@15.0.0_react-dom@19.0.0-rc-65a56d0e-20241020_react@19.0.0-rc-65a56d0e-20241020__react@19.0.0-rc-65a56d0e-20241020/node_modules/next/dist/build/utils.js:1233:15 at process.processTicksAndRejections (node:internal/process/task_queues:105:5) { type: 'Error' } Collecting page data . ELIFECYCLE Command failed with exit code 1. ```

Provide environment information

Operating System:
  Platform: linux
  Arch: x64
  Version: #1-NixOS SMP PREEMPT_DYNAMIC Thu Aug 29 15:33:59 UTC 2024
  Available memory (MB): 31938
  Available CPU cores: 20
Binaries:
  Node: 22.8.0
  npm: N/A
  Yarn: N/A
  pnpm: 9.10.0
Relevant Packages:
  next: 15.0.0 // Latest available version is detected (15.0.0).
  eslint-config-next: N/A
  react: 18.3.1
  react-dom: 18.3.1
  typescript: 5.6.2
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

Not sure

Which stage(s) are affected? (Select all that apply)

next dev (local), next build (local)

Additional context

The same setup compiles when using Next.js v14.2.13 for web-app. It seems that the bug is introduced in v14.2.14.

lubieowoce commented 1 month ago

Hmm, pretty weird, looks like some kind of path resolution issue

import { ... } from 'rust-lib' is somehow resolved to a path that's relative to packages/web-app, the project root (require("../rust-lib/dist/index.node"). But then that require() call is placed in packages/web-app/.next/server/app/page.js, so the ../ ends up being incorrect... Manually editing page.js to do require("../../../../rust-lib/dist/index.node") fixes the problem.

lubieowoce commented 3 weeks ago

This still seems to be a problem in 15.0.2, but in a different way.

We've removed the code that was resolving paths incorrectly in #71723, so the original error is no longer reproducible.

But now, it's failing with

Module parse failed: Unexpected character '�' (1:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
(Source code omitted for this binary file)

Import trace for requested module:
../rust-lib/dist/index.node
./src/app/page.tsx
 ○ Compiling /_error ...
 ⨯ ../rust-lib/dist/index.node

...and making the package external doesn't seem to have any effect, which is pretty suspicious:

const nextConfig: NextConfig = {
  serverExternalPackages: ["rust-lib"],
};

this makes me suspect that something's buggy with serverExternalPackages that point to local packages

(edit: i think this is because in our optOutBundlingPackageRegex we assume that the resolved path contains /node_modules/, but "rust-lib" resolves to a path that doesn't have it. because the node_modules symlink for rust-lib is resolved to its real path)

vercel / next.js