search

vercel / nft

Node.js dependency tracing utility
https://npmjs.com/@vercel/nft
MIT License
1.31k stars 136 forks source link

Add support for `zeromq` #391

Closed eulersson closed 7 months ago

eulersson commented 7 months ago

Some libraries like zeromq.js need runtime libraries with bindings which they ship along. Those binaries exist under node_modules/zeromq/prebuilds.

Simply by creating a new Next.js app, installing https://github.com/zeromq/zeromq.js and trying to import it and running npm run dev and use one of its classes would throw a No native build was found for platform=linux arch=x64 runtime=node abi=115 uv=1 libc=glibc node=20.11.0 webpack=true loaded from: /app/.next/server. I found out that if I print this dir variable used (among other things) to search for binaries in node-gyp-builder it print .next/server/... instead of node_modules/... and since the binaries have not been copied it fails.

[!NOTE] I provided a reproducible example and a container, but it's easily reproducible on mac and linux: https://github.com/eulersson/zeromq.js-next.js-errors

I tried using the outputFileTracingIncludes option as explained in https://nextjs.org/docs/pages/api-reference/next-config-js/output#caveats but without success.

Here's the files I would like to have in .next/server.

❯ tree node_modules/zeromq/prebuilds
node_modules/zeromq/prebuilds
├── darwin-arm64
│   └── node.napi.glibc.node
├── darwin-x64
│   └── node.napi.glibc.node
├── linux-x64
│   ├── node.napi.glibc.node
│   └── node.napi.musl.node
├── win32-ia32
│   └── node.napi.glibc.node
└── win32-x64
    └── node.napi.glibc.node

6 directories, 6 files

Since I don't know in what area work should be done I posted the same problem as an issue to the various tools involved:

Thanks for the help and apologies if I have been generating noise by posting so many issues.

styfle commented 7 months ago

You can try npx @vercel/nft@latest print path/to/my/file.js to see which files are included.

If its missing prebuilds, then it is likely an issue with @vercel/nft.

You can create a PR with a test and special case, similar to PR https://github.com/vercel/nft/pull/37.

Or if you want to fix the general case (likely a dynamic require somewhere) that would be great too (albeit much harder).

eulersson commented 7 months ago

@styfle Many thanks for the guidance. I will investigate that route this afternoon! :)

eulersson commented 7 months ago

@styfle Indeed it certainly does not bring in the binaries:

❯ npx @vercel/nft@latest print ./node_modules/zeromq/lib/index.js
FILELIST:
node_modules/@aminya/node-gyp-build/index.js
node_modules/@aminya/node-gyp-build/package.json
node_modules/zeromq/lib/draft.js
node_modules/zeromq/lib/index.js
node_modules/zeromq/lib/native.js
node_modules/zeromq/lib/util.js
node_modules/zeromq/package.json
eulersson commented 7 months ago

@styfle I will provide a PR tomorrow.

styfle commented 7 months ago

Also see this related PR that mentions node-gyp-build

That might lead to the more general purpose fix instead of a special case.

eulersson commented 7 months ago

I will do the general purpose one. I think I understood the tracing logic a bit.

eulersson commented 7 months ago

@styfle Can you have a look? 👀 🙇

github-actions[bot] commented 7 months ago

:tada: This issue has been resolved in version 0.26.4 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

eulersson commented 7 months ago

@styfle how can I test this? I am trying to use the overrides npm feature without success:

https://github.com/eulersson/zeromq.js-next.js-errors/blob/b0e24cbf471d5f454cbe3f6b89f7083d0027474a/package.json#L28-L30

I also set next and eslint-config-next to ^14.1.1-canary.52 which seems to have the @vercel/nft to 0.26.4 but in reality, if after npm installing I check node_modules/next/package.json I see 0.26.3 and not 0.26.4.

I'm very confused because at this commit https://github.com/vercel/next.js/blob/07c652a1200445ebfed8ef13426e36a150a04a67/packages/next/package.json#L194 (Next.js 14.1.1-canary.52) I can see the nft version set at 0.26.4.

Can you help me set up a project with a next app that has my changes?

eulersson commented 7 months ago

@styfle I can see the tagged next git repo 14.1.1-canary.52 does have 0.26.3 but the canary branch carrying the same package.json version carries 0.26.4. I was thinking of using something like this:

  "dependencies": {
    "next": "https://github.com/vercel/next.git#0c21654845f78717a8198a69d968671477adc7f8",

0c21654845f78717a8198a69d968671477adc7f8 is the commit of the canary branch that has the changes I want.

Would that work?

The next.js repo has many packages: next, eslint-config-next, etc... I don't know if putting a git url in packages.json is the best solution for testing my changes....

styfle commented 7 months ago

@eulersson A new canary was just released with the fix. You can run npm i next@14.1.1-canary.53 to try it out now.

https://github.com/vercel/next.js/releases/tag/v14.1.1-canary.53

eulersson commented 7 months ago

Still not working @aminya @styfle. Do you have any idea what might be the issue?

zeromq.js-next.js-errors git:main* 
❯ npm run build
> my-app@0.1.0 build
> next build

   ▲ Next.js 14.1.1-canary.53

   Creating an optimized production build ...

warn - No utility classes were detected in your source files. If this is unexpected, double-check the `content` option in your Tailwind CSS configuration.
warn - https://tailwindcss.com/docs/content-configuration
 ✓ Compiled successfully
 ✓ Linting and checking validity of types
   Collecting page data  .Error: No native build was found for platform=darwin arch=x64 runtime=node abi=115 uv=1 libc=glibc node=20.11.0 webpack=true
    loaded from: /Users/ramon/Playground/zeromq.js-next.js-errors/.next/server

    at g.path (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:1:5427)
    at g (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:1:3762)
    at 9140 (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:6:36323)
    at t (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/webpack-runtime.js:1:127)
    at 6874 (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:6:29810)
    at t (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/webpack-runtime.js:1:127)
    at 1100 (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/app/page.js:1:2991)
    at Function.t (/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server/webpack-runtime.js:1:127)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async collectGenerateParams (/Users/ramon/Playground/zeromq.js-next.js-errors/node_modules/next/dist/build/utils.js:920:21)

> Build error occurred
Error: Failed to collect page data for /
    at /Users/ramon/Playground/zeromq.js-next.js-errors/node_modules/next/dist/build/utils.js:1268:15
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  type: 'Error'
}
   Collecting page data  .%                                                                                                                    zeromq.js-next.js-errors git:main*  15s

However the nft version is correct and brings in the binary.

❯ npx @vercel/nft print ./node_modules/zeromq/lib/index.js
FILELIST:
node_modules/@aminya/node-gyp-build/index.js
node_modules/@aminya/node-gyp-build/package.json
node_modules/zeromq/lib/draft.js
node_modules/zeromq/lib/index.js
node_modules/zeromq/lib/native.js
node_modules/zeromq/lib/util.js
node_modules/zeromq/package.json
node_modules/zeromq/prebuilds/darwin-x64/node.napi.glibc.node

It breaks here at the throw error:

https://github.com/aminya/node-gyp-build/blob/4cd1eb0f1ef2ed7d4c36b82e3d547e33cd83c9a0/index.js#L25-L60

The variable dir in line 26 is evaluated to '/Users/ramon/Playground/zeromq.js-next.js-errors/.next/server when debugging, this is what the folder looks like:

zeromq.js-next.js-errors/.next/server git:main*
❯ tree
.
├── app
│   ├── _not-found_client-reference-manifest.js
│   ├── not-found.js
│   ├── not-found_client-reference-manifest.js
│   ├── page.js
│   └── page_client-reference-manifest.js
├── app-paths-manifest.json
├── interception-route-rewrite-manifest.js
├── middleware-build-manifest.js
├── middleware-manifest.json
├── middleware-react-loadable-manifest.js
├── next-font-manifest.js
├── next-font-manifest.json
├── pages-manifest.json
├── server-reference-manifest.js
├── server-reference-manifest.json
├── vendor-chunks
│   ├── @aminya.js
│   ├── @swc.js
│   ├── next.js
│   └── zeromq.js
└── webpack-runtime.js

3 directories, 20 files

I will attach here the contents of .next/server/vendor-chunks/zeromq.js. zeromq.js.txt

To reproduce simply import zeromq@6.0.0-beta.19 into your next canary app greater than 14.1.1-canary.53 and place import { Push } from "zeromq"; and const socket = new Push() anywhere .

styfle commented 7 months ago

This will fix it:

eulersson commented 7 months ago

@styfle I confirm it works now! Thanks for your help 🙏