vercel / nft

Node.js dependency tracing utility
https://npmjs.com/@vercel/nft
MIT License
1.31k stars 136 forks source link

fix: support datadog-pprof #411

Closed austinwoon closed 4 months ago

austinwoon commented 5 months ago

Closes #410

Alternative solution considered

Solution

Instead of the alternative above, chose to just add a special case for this package

Automated tests

Just added a conditional testName === '@datadog-pprof' to the foundMatchingBinary statement so we test the presence of the prebuilds folder.

Manual tests (since the automated tests do not test for presence of prebuilds folder

Running node out/cli.js print node_modules/@datadog/pprof/out/src/index.js gives me the following FILELIST

FILELIST:
node_modules/@datadog/pprof/node_modules/node-gyp-build/index.js
node_modules/@datadog/pprof/node_modules/node-gyp-build/package.json
node_modules/@datadog/pprof/node_modules/source-map/lib/array-set.js
node_modules/@datadog/pprof/node_modules/source-map/lib/base64-vlq.js
node_modules/@datadog/pprof/node_modules/source-map/lib/base64.js
node_modules/@datadog/pprof/node_modules/source-map/lib/binary-search.js
node_modules/@datadog/pprof/node_modules/source-map/lib/mapping-list.js
node_modules/@datadog/pprof/node_modules/source-map/lib/mappings.wasm
node_modules/@datadog/pprof/node_modules/source-map/lib/read-wasm.js
node_modules/@datadog/pprof/node_modules/source-map/lib/source-map-consumer.js
node_modules/@datadog/pprof/node_modules/source-map/lib/source-map-generator.js
node_modules/@datadog/pprof/node_modules/source-map/lib/source-node.js
node_modules/@datadog/pprof/node_modules/source-map/lib/util.js
node_modules/@datadog/pprof/node_modules/source-map/lib/wasm.js
node_modules/@datadog/pprof/node_modules/source-map/package.json
node_modules/@datadog/pprof/node_modules/source-map/source-map.js
node_modules/@datadog/pprof/out/src/heap-profiler-bindings.js
node_modules/@datadog/pprof/out/src/heap-profiler.js
node_modules/@datadog/pprof/out/src/index.js
node_modules/@datadog/pprof/out/src/logger.js
node_modules/@datadog/pprof/out/src/profile-encoder.js
node_modules/@datadog/pprof/out/src/profile-serializer.js
node_modules/@datadog/pprof/out/src/sourcemapper/sourcemapper.js
node_modules/@datadog/pprof/out/src/time-profiler-bindings.js
node_modules/@datadog/pprof/out/src/time-profiler.js
node_modules/@datadog/pprof/package.json
node_modules/@datadog/pprof/prebuilds/darwin-arm64/node-102.node
node_modules/@datadog/pprof/prebuilds/darwin-arm64/node-108.node
node_modules/@datadog/pprof/prebuilds/darwin-arm64/node-111.node
node_modules/@datadog/pprof/prebuilds/darwin-arm64/node-115.node
node_modules/@datadog/pprof/prebuilds/darwin-arm64/node-120.node
node_modules/@datadog/pprof/prebuilds/darwin-arm64/node-83.node
node_modules/@datadog/pprof/prebuilds/darwin-arm64/node-88.node
node_modules/@datadog/pprof/prebuilds/darwin-arm64/node-93.node
node_modules/@datadog/pprof/prebuilds/darwin-x64/node-102.node
node_modules/@datadog/pprof/prebuilds/darwin-x64/node-108.node
node_modules/@datadog/pprof/prebuilds/darwin-x64/node-111.node
node_modules/@datadog/pprof/prebuilds/darwin-x64/node-115.node
node_modules/@datadog/pprof/prebuilds/darwin-x64/node-120.node
node_modules/@datadog/pprof/prebuilds/darwin-x64/node-83.node
node_modules/@datadog/pprof/prebuilds/darwin-x64/node-88.node
node_modules/@datadog/pprof/prebuilds/darwin-x64/node-93.node
node_modules/@datadog/pprof/prebuilds/linux-arm64/node-102.node
node_modules/@datadog/pprof/prebuilds/linux-arm64/node-108.node
node_modules/@datadog/pprof/prebuilds/linux-arm64/node-111.node
node_modules/@datadog/pprof/prebuilds/linux-arm64/node-115.node
node_modules/@datadog/pprof/prebuilds/linux-arm64/node-120.node
node_modules/@datadog/pprof/prebuilds/linux-arm64/node-83.node
node_modules/@datadog/pprof/prebuilds/linux-arm64/node-88.node
node_modules/@datadog/pprof/prebuilds/linux-arm64/node-93.node
node_modules/@datadog/pprof/prebuilds/linux-x64/node-102.node
node_modules/@datadog/pprof/prebuilds/linux-x64/node-108.node
node_modules/@datadog/pprof/prebuilds/linux-x64/node-111.node
node_modules/@datadog/pprof/prebuilds/linux-x64/node-115.node
node_modules/@datadog/pprof/prebuilds/linux-x64/node-120.node
node_modules/@datadog/pprof/prebuilds/linux-x64/node-83.node
node_modules/@datadog/pprof/prebuilds/linux-x64/node-88.node
node_modules/@datadog/pprof/prebuilds/linux-x64/node-93.node
node_modules/@datadog/pprof/prebuilds/linuxmusl-x64/node-102.node
node_modules/@datadog/pprof/prebuilds/linuxmusl-x64/node-108.node
node_modules/@datadog/pprof/prebuilds/linuxmusl-x64/node-111.node
node_modules/@datadog/pprof/prebuilds/linuxmusl-x64/node-115.node
node_modules/@datadog/pprof/prebuilds/linuxmusl-x64/node-120.node
node_modules/@datadog/pprof/prebuilds/linuxmusl-x64/node-83.node
node_modules/@datadog/pprof/prebuilds/linuxmusl-x64/node-88.node
node_modules/@datadog/pprof/prebuilds/linuxmusl-x64/node-93.node
node_modules/@datadog/pprof/prebuilds/win32-ia32/node-102.node
node_modules/@datadog/pprof/prebuilds/win32-ia32/node-83.node
node_modules/@datadog/pprof/prebuilds/win32-ia32/node-88.node
node_modules/@datadog/pprof/prebuilds/win32-ia32/node-93.node
node_modules/@datadog/pprof/prebuilds/win32-x64/node-102.node
node_modules/@datadog/pprof/prebuilds/win32-x64/node-108.node
node_modules/@datadog/pprof/prebuilds/win32-x64/node-111.node
node_modules/@datadog/pprof/prebuilds/win32-x64/node-115.node
node_modules/@datadog/pprof/prebuilds/win32-x64/node-120.node
node_modules/@datadog/pprof/prebuilds/win32-x64/node-83.node
node_modules/@datadog/pprof/prebuilds/win32-x64/node-88.node
node_modules/@datadog/pprof/prebuilds/win32-x64/node-93.node
node_modules/delay/index.js
node_modules/delay/package.json
node_modules/p-limit/index.js
node_modules/p-limit/package.json
node_modules/pprof-format/dist/index.js
node_modules/pprof-format/package.json
node_modules/yocto-queue/index.js
node_modules/yocto-queue/package.json

We see that the entire prebuilds folder is correctly included.

styfle commented 5 months ago

Looks like we're hitting this NODE_GYP_BUILD case correctly, however the AST is not understood and so node-gyp-build is never executed.

https://github.com/vercel/nft/blob/c16e7945c3f3221616567356d9abc2e824bf38cd/src/analyze.ts#L755

austinwoon commented 5 months ago

Looks like we're hitting this NODE_GYP_BUILD case correctly, however the AST is not understood and so node-gyp-build is never executed.

https://github.com/vercel/nft/blob/c16e7945c3f3221616567356d9abc2e824bf38cd/src/analyze.ts#L755

Yes, I described why this was the case in the issue here

Datadog is using a CJS import and assigning it to a variable. So in the AST, we will not be able to infer the package name from the arguments (the arguments are findBinding, .., .. respectively, instead of node-gyp-build, .. or node-gyp-build, .., .. like in the other packages this statement was made for)

I also find that having to manually resolve the arguments via checking the array of args a little hard-codey, so i decided to just have a special case catch all instead for the package to just include prebuilds as it seemed simpler. Hope that makes sense!

austinwoon commented 5 months ago

Looks like we're hitting this NODE_GYP_BUILD case correctly, however the AST is not understood and so node-gyp-build is never executed.

https://github.com/vercel/nft/blob/c16e7945c3f3221616567356d9abc2e824bf38cd/src/analyze.ts#L755

Could I also check if it would be possible to include a flag to ignore tree shaking modules in a package? I think it would make sense as a catch-all to handle special case like these!

austinwoon commented 5 months ago

I think we can revisit the general solution in a new PR and merge this PR once the windows test pass

I have removed the sourcemaps outputs from output.js since there is no actual need to test for those folders. Could you please rerun the test pipeline? Thank you

styfle commented 4 months ago

I took a stab at fixing this properly in PR https://github.com/vercel/nft/pull/419