pkg packing uneeded build files for native modules

[Mikescops]

What version of pkg are you using?

5.11.5

What version of Node.js are you using?

18.x

What operating system are you using?

macOS

What CPU architecture are you using?

M2

What Node versions, OSs and CPU architectures are you building for?

any

Describe the Bug

Hello,

I'm trying to optimize how my native modules are loaded into the virtual filesystem. I know I could use esbuild, but so far I've only came across hacky ways of bundling .node with it, because there are several ways of doing them between bindings, node-gyp, napi-rs...

The way pkg does it seems fair to me and is less likely to create side effects as it follows the same structure as the node_modules folder.

My issue is that for better-sqlite3, that is working well when packaged with pkg, I end up with all the build files (cpp and c) being added in the final VFS. (https://github.com/WiseLibs/better-sqlite3)

If I use the debug command to explore the VFS, this is what I see:

    node_modules                           17.8 MB 
      better-sqlite3                       11.3 MB 
        package.json                        1.5 KB 
        binding.gyp                      943 Bytes 
        lib                                22.7 KB 
          index.js                       110 Bytes 
          database.js                       4.0 KB 
          sqlite-error.js                717 Bytes 
          util.js                        331 Bytes 
          methods                          17.6 KB 
            aggregate.js                    1.9 KB 
            backup.js                       2.3 KB 
            function.js                     1.4 KB 
            inspect.js                   174 Bytes 
            pragma.js                    536 Bytes 
            serialize.js                 625 Bytes 
            table.js                        7.0 KB 
            transaction.js                  2.6 KB 
            wrappers.js                     1.1 KB 
        src                               170.6 KB 
          better_sqlite3.cpp              127.8 KB 
          better_sqlite3.hpp               42.8 KB 
        deps                                9.3 MB 
          common.gypi                       1.5 KB 
          copy.js                        897 Bytes 
          defines.gypi                      1.1 KB 
          download.sh                       3.3 KB 
          sqlite3.gyp                       2.5 KB 
          test_extension.c               588 Bytes 
          sqlite3                           9.3 MB 
            sqlite3.c                       8.7 MB 
            sqlite3.h                     629.0 KB 
            sqlite3ext.h                   37.3 KB 
        build                               1.8 MB 
          Release                           1.8 MB 
            better_sqlite3.node             1.8 MB 

When looking at the debug logs I figured out that the files are being included because when pkg reads the package.json it sees this:

  "main": "lib/index.js",
  "files": [
    "binding.gyp",
    "src/*.[ch]pp",
    "lib/**",
    "deps/**"
  ],

Expected Behavior

The builds files (.c and .cpp) could be excluded from the final build.

I'm not sure what's the best solution here:

• being able to edit the included files per package
• having some block list of file extensions
• being able to remove glob patterns from the assets list

To Reproduce

Create a sample project that uses better-sqlite3 or use our open source project that uses it: https://github.com/Dashlane/dashlane-cli

[robertsLando]

pkg tries to do it's best to guess the files he needs but this is hard to do so I don't see many ways to improve this on pkg side. What you could do is to create a script to do the package and simply move away files before the pkg process and move them back after it finishes

[Mikescops]

Don't you think blocked glob patterns could be a nice way to handle this? That way when the analyzer encounters a file it checks it against the block list and ignore the files if it matches one of the patterns.

[robertsLando]

Yeah but I have no time to implement such feature right now. Want to submit a PR?

I would call it like ignore and will be an array of glob like scripts and assets

[Mikescops]

I can give it a try when I have some times too yes, ignore seems fair yep

[segevfiner]

Still have cases with packages using prebuildify packaging the binaries for all platforms instead of just the built platform, should I open another issue about that?

[robertsLando]

Actually this is due to https://github.com/yao-pkg/pkg/blob/main/prelude/bootstrap.js#L2235

Reason described in the comment above that line and I think there is no easy fix for that

See ref issue: https://github.com/vercel/pkg/issues/1075