Support for native `.node` modules

[Kinrany]

I'm trying to use esbuild to minify my Node project, to make my container images smaller with a multi-stage build.

I need lovell/sharp, which has a .node module. This breaks the build.

I could mark that module as external. But I'm also using pnpm, so the package and its dependencies are actually behind symlinks. It seems I'd have to manually move modules and replace paths in esbuild output to make this work.

Ideally esbuild would assume that native modules have no other dependencies and just place them next to the regular output.

[evanw]

You could argue that esbuild should handle this itself. However, it currently doesn't do this. The file loader almost does this but it returns a path to the file instead of loading the file. But you can use the file loader in a small plugin to do what you want without needing this feature to be built into esbuild:

const nativeNodeModulesPlugin = {
  name: 'native-node-modules',
  setup(build) {
    // If a ".node" file is imported within a module in the "file" namespace, resolve 
    // it to an absolute path and put it into the "node-file" virtual namespace.
    build.onResolve({ filter: /\.node$/, namespace: 'file' }, args => ({
      path: require.resolve(args.path, { paths: [args.resolveDir] }),
      namespace: 'node-file',
    }))

    // Files in the "node-file" virtual namespace call "require()" on the
    // path from esbuild of the ".node" file in the output directory.
    build.onLoad({ filter: /.*/, namespace: 'node-file' }, args => ({
      contents: `
        import path from ${JSON.stringify(args.path)}
        try { module.exports = require(path) }
        catch {}
      `,
    }))

    // If a ".node" file is imported within a module in the "node-file" namespace, put
    // it in the "file" namespace where esbuild's default loading behavior will handle
    // it. It is already an absolute path since we resolved it to one above.
    build.onResolve({ filter: /\.node$/, namespace: 'node-file' }, args => ({
      path: args.path,
      namespace: 'file',
    }))

    // Tell esbuild's default loading behavior to use the "file" loader for
    // these ".node" files.
    let opts = build.initialOptions
    opts.loader = opts.loader || {}
    opts.loader['.node'] = 'file'
  },
}

[evanw]

I tested the above plugin with the fsevents module and it worked for that. So the plugin does work for some packages with .node files.

But from the discussion in #972, it sounds like lovell/sharp won't work even if .node files are copied over because that package also needs to load other random files from the file system as determined by run-time environment variables, which cannot be bundled ahead of time (i.e. the sharp package is incompatible with bundling).

So the solution here is to mark this module as external with --external:sharp and make sure the sharp package is still installed at run-time when you run your bundle.

[Kinrany]

Would it make sense for esbuild to copy packages that shouldn't be bundled into a new node_modules in the output directory?

Perhaps even minify each package, but keep the separation between packages.

The problem I'm still having in my case is that node_modules only contains symlinks, and I'd have to resolve all of those to make sure all necessary code is inside outdir.

[deadcoder0904]

@evanw Would love a solution to this as well. I'm using mdx-bundler which uses esbuild underhood. I run sharp to convert gif files to png & move them to the public/ folder.

So tried creating a plugin, see https://github.com/kentcdodds/mdx-bundler/issues/74

But it yells at using .node

[fr-an-k]

+1

[iammati]

Noticed that one of my dependencies (limax - https://github.com/lovell/limax) also uses .node but it isn't supported by esbuild yet. Is there at least a workaround for this? Development is a struggle without a dev-server 🤣

EDIT: welp, a workaround (at least in my case) was easier than I thought (using vitejs!):

import { UserConfig, defineConfig } from 'vite'

const configuration: UserConfig = {
    ...
    optimizeDeps: {
        exclude: [
            'limax',
        ],
    },
}

export default defineConfig(configuration)

Excluding the package/dependency which has a .node file and throws something like error: No loader is configured for ".node" files: simply add the snippet with the optimizeDeps.exclude inside your vite.config.(js|ts) file and you should be able to use the dev-server again.

Also see: https://vitejs.dev/config/#optimizedeps-exclude

I hope for anyone who stumbles over this issue can make an use of this (:

[jeffgunderson]

I get this same thing in a Jenkins environment with ssh2 library using serverless framework + serverless-esbuild plugin. I fixed it by adding it to the external list

esbuild: {
    plugins: 'esbuild.plugins.js',
    external: ['pg-native', 'ssh2'],
},

[spion]

Would it be okay if we added additional options to the onLoad / onResolve callback, specifying additional files to include? That way onResolve could return any additional .dll / .so /.other` files that would need to be copied.

[shivangsanghi]

I

I am getting error: Cannot use the "file" loader without an output path

[evanw]

With #2320 in the upcoming release, you should now be able to copy these files over. So I'm going to close this issue as resolved. This will likely only work for a few native node modules though. Most of the time you are probably better off just marking the whole package as external because these packages typically access their package directory on the file system, which doesn't work from inside a bundle.

[tiberiuzuld]

Hello, Copy loader didn't work for me https://github.com/evanw/esbuild/pull/2320 for the sharp library. Instead I found a workaround by doing a string replace in the output bundle and copying the ./node_modules/sharp/build/Release/ folder to the output folder.

import fs from 'fs/promises';
import replace from 'replace-in-file';
  // esbuild
  .then(() =>
    replace({
      files: './out/index.js',
      from: '../build/Release/sharp-',
      to: './sharp-'
    })
  )
  .then(() =>
    fs.cp('./node_modules/sharp/build/Release/', './out', {
      recursive: true
    })
  )

It's the most simple workaround I could find that works.

[vaunus]

You're a life saver @tiberiuzuld! I had given up trying to get this working this morning and was swapping to another lib when I spotted your reply. I managed to get sharp working on AWS Lambda by doing the above, ie editing my handler.js and fixing the path as you describe, then copying the sharp .node binary in. I also needed to copy libvips-cpp.so.42 and it's associated versions.json in as well and edited their paths as they contain ../vendor. I have done this manually for now as a POC and it works but now need to formalise it as part of our build process.

[BonnieMilianB]

thanks for the idea @tiberiuzuld I was having the same issue.

If useful to somebody, what I did was a script to copy sharp and the libs that sharp needs, to my build.

So on package.json

"scripts": {
    ...
    "build": "esbuild --bundle src/index.ts --external:sharp --platform=node --target=node16 --outdir=build/",
    "build:externals": "ts-node bundleSharp.ts",
}

And the bundleSharp.ts script:

import fs from 'fs/promises'

const copySharp = async () => {
  await fs.cp('./node_modules/sharp', './build/node_modules/sharp', {
    recursive: true
  })
  await fs.cp('./node_modules/color', './build/node_modules/color', {
    recursive: true
  })
  await fs.cp('./node_modules/color-convert', './build/node_modules/color-convert', {
    recursive: true
  })
  await fs.cp('./node_modules/color-name', './build/node_modules/color-name', {
    recursive: true
  })
  await fs.cp('./node_modules/color-string', './build/node_modules/color-string', {
    recursive: true
  })
  await fs.cp('./node_modules/detect-libc', './build/node_modules/detect-libc', {
    recursive: true
  })
  await fs.cp('./node_modules/is-arrayish', './build/node_modules/is-arrayish', {
    recursive: true
  })
  await fs.cp('./node_modules/semver', './build/node_modules/semver', {
    recursive: true
  })
  await fs.cp('./node_modules/simple-swizzle', './build/node_modules/simple-swizzle', {
    recursive: true
  })
}

copySharp()

[swyxio]

chiming in to note that this plugin did not work for @resvg/resvg-js: https://github.com/yisibl/resvg-js/issues/175

[evanw]

Native node modules could very well depend on the location of files in the file system. In that case no plugin is going to work. You'll have to mark the package as external to exclude it from the bundle, and make sure the files are on the file system in the right places at run-time when the bundle is evaluated.

[LinirZamir]

Hey did you figure out a solution to this issue? I am having the same problem

[dallen4]

@LinirZamir the --loader flag used in this comment works for me.

[oleksandr-danylchenko]

@LinirZamir the --loader flag used in this comment works for me.

What comment?

[Aarbel]

On my side using --loader:.node=file solved the problem, but it was hard to set because i thought it was applied to another deploy / environnement.

On AWS cdk i did that:

          bundling: {
            externalModules: props.nodeModules.dependencies,
            loader: {
              ".node": "file",
            },
            minify: true,
            sourceMap: true,
          },

[felixebert]

For those finding this issue when trying to use sharp in a NodeJS Lambda Function (like me):

I can successfully use sharp in a lambda nodejs CDK function using esbuild bundling (not docker bundling & no lambda layer) with the following options. Also works in a Github Action workflow.

(Note: Use --arch=arm64 for CDK lambda.Architecture.ARM_64

new lambdaNodejs.NodejsFunction(this, "SOME_ID", {
  // ...
  bundling: {
    externalModules: ["sharp"],
    nodeModules: ["sharp"],
    commandHooks: {
      // eslint-disable-next-line @typescript-eslint/no-unused-vars
      beforeBundling(inputDir: string, outputDir: string): string[] {
        return [];
      },
      // eslint-disable-next-line @typescript-eslint/no-unused-vars
      beforeInstall(inputDir: string, outputDir: string): string[] {
        return [];
      },
      afterBundling(inputDir: string, outputDir: string): string[] {
        return [`cd ${outputDir}`, "rm -rf node_modules/sharp && npm install --no-save --arch=x86 --platform=linux sharp"];
      }
    }
  }
  // ...
});

Taken from aws-solutions/serverless-image-handler

[robertsLando]

Any clue how I can make this work with serialport?

[nopol10]

For those using serverless-esbuild and encountering this with ssh2 (or probably other packages), instead of adding ssh2 to external, you can add the loader option to esbuild's settings like this:

serverless.yml:

esbuild:
  ...
  loader:
    ".node": "file"

This will cause the sshcrypto.node file to be copied to the same folder as the js file generated by esbuild and the resulting js should point to the copied file's path correctly.

This is if you don't have ssh2 in the lambda's environment (which some people might have in a layer).

[artiz]

This config works for me:

/* eslint-disable */
/* tslint:disable */
const { build } = require("esbuild");
const path = require("path");
const fs = require("fs");

const findBinaryFiles = (dir) => {
   const binaries = [];
   const files = fs.readdirSync(dir);
   for (const file of files) {
      const filePath = path.join(dir, file);
      const stat = fs.statSync(filePath);
      if (stat.isDirectory()) {
         binaries.push(...findBinaryFiles(filePath));
      } else if (path.extname(file) === ".node") {
         binaries.push(filePath);
      }
   }
   return binaries;
};

const nativeNodeModulesPlugin = {
   name: "native-node-modules",
   setup(build) {
      const baseOutdir = build.initialOptions.outdir || path.dirname(build.initialOptions.outfile);
      const outdir = path.resolve(baseOutdir);
      const buildDir = path.join(outdir, 'build');

      if (!fs.existsSync(outdir)) fs.mkdirSync(outdir);
      if (!fs.existsSync(buildDir)) fs.mkdirSync(buildDir);

      const processedBinaries = new Set();

      build.onResolve({ filter: /bindings/, namespace: "file" }, (args) => {
         const filePath =  require.resolve(args.path, { paths: [args.resolveDir] });
         const { resolveDir } = args;
         let packageDir = path.dirname(resolveDir);
         while(packageDir && path.basename(packageDir) !== "node_modules") {
            packageDir = path.dirname(packageDir);
         }
         packageDir = path.dirname(packageDir);

         // find '.node' files in the packageDir
         const binaries = findBinaryFiles(packageDir);
         binaries.forEach((binary) => {
            const fname = path.basename(binary);
            if (!processedBinaries.has(fname)) {
               const outPath = path.join(buildDir, fname);
               fs.copyFileSync(binary, outPath);
               processedBinaries.add(fname);
            }
         });

         return {
            path: filePath,
            namespace: "bindings",
         };
      });

      build.onLoad({ filter: /.*/, namespace: "bindings" }, (args) => {
         return {
            contents: `
            const path = require("path");
            const fs = require("fs");
            const __bindings = require(${JSON.stringify(args.path)});

            module.exports = function(opts) {
               if (typeof opts == "string") {
                  opts = { bindings: opts };
               } else if (!opts) {
                  opts = {};
               }

               opts.module_root = path.dirname(__filename);
               return __bindings(opts);
            };
          `,
         };
      });

      build.onResolve({ filter: /bindings\.js$/, namespace: "bindings" }, (args) => {
         return {
            path: args.path,
            namespace: "file",
         };
      });
   },
};

const options = {
   entryPoints: ["src/index.ts"],
   outfile: "deploy/index.js",
   bundle: true,
   minify: false,
   sourcemap: true,
   external: ["aws-sdk"],
   platform: "node",
   loader: {
      ".svg": "file",
      ".html": "text",
   },
   define: {},
   plugins: [nativeNodeModulesPlugin],
};

build(options).catch(() => process.exit(1));

[justinwaite]

For those using serverless-esbuild and encountering this with ssh2 (or probably other packages), instead of adding ssh2 to external, you can add the loader option to esbuild's settings like this:

serverless.yml:

esbuild:
  ...
  loader:
    ".node": "file"

This will cause the sshcrypto.node file to be copied to the same folder as the js file generated by esbuild and the resulting js should point to the copied file's path correctly.

This is if you don't have ssh2 in the lambda's environment (which some people might have in a layer).

Worked for me when deploying an SST Ion Remix app with the react-pdf package.

    new sst.aws.Remix('MyApp', {
      transform: {
        server: {
          nodejs: {
            esbuild: {
              loader: {
                '.node': 'file',
              },
            },
          },
        },
      },
    });