Webpack 5 breaks dynamic wasm import for SSR

[TimoWilhelm]

What version of Next.js are you using?

10.2.3

What version of Node.js are you using?

14.16.0

What browser are you using?

Chrome

What operating system are you using?

Windows

How are you deploying your application?

Other

Describe the Bug

Using Webpack 5 breaks dynamic import for WASM modules when using SSR.

ENOENT: no such file or directory, open '...\.next\server\static\wasm

I've provided a minimal reproducible example here: https://github.com/TimoWilhelm/mre-next-with-webassembly-webpack-5

Expected Behavior

Dynamic import of WASM modules should work for SSR when using Webpack 5.

To Reproduce

Run npm run build

info  - Using webpack 5. Reason: future.webpack5 option enabled https://nextjs.org/docs/messages/webpack5
info  - Checking validity of types
info  - Creating an optimized production build  
info  - Compiled successfully
info  - Collecting page data  
[=== ] info  - Generating static pages (0/3)
Error occurred prerendering page "/". Read more: https://nextjs.org/docs/messages/prerender-error
Error: ENOENT: no such file or directory, open 'C:\...\with-webassembly\.next\server\static\wasm\1f565eb157746630b627.wasm'
info  - Generating static pages (3/3)

Disabling SSR for the dynamic import fixes the issue.

 const RustComponent = dynamic({
   loader: async () => {
     // Import the wasm module
     const rustModule = await import('../add.wasm')
     // Return a React component that calls the add_one method on the wasm module
     return (props) => <div>{rustModule.add_one(props.number)}</div>
   },
+  ssr: false,
 });

Downgrading the Webpack version to use v4 also fixes the issue.

 module.exports = {
   future: {
-    webpack5: true,
+    webpack5: false,
   },
   webpack(config) {
-    config.experiments = { syncWebAssembly: true };
     config.output.webassemblyModuleFilename = "static/wasm/[modulehash].wasm";
     return config;
   },
...

[TimoWilhelm]

It seems like the issue is the following line part of the webpack config:

https://github.com/vercel/next.js/blob/1ebf26af784637a27fe422090418126c474353f4/packages/next/build/webpack-config.ts#L918-L921

The server output gets prefixed with 'chunks' which can't be resolved by the static site generation.

I've managed to make it work by using the following webassemblyModuleFilename config in my next.config.js but that doesn't seem ideal.

webpack: (config, { isServer }) => {
  config.experiments = { asyncWebAssembly: true };

  if (isServer) {
    config.output.webassemblyModuleFilename =
      './../static/wasm/[modulehash].wasm';
  } else {
    config.output.webassemblyModuleFilename =
      'static/wasm/[modulehash].wasm';
  }

  return config;
},

[franky47]

@TimoWilhelm I tried your hack but it did not work: webpack is trying to import a WASM file with a different hash than the ones that are copied to .next/static/wasm, even though I have a single WASM file in my whole project. Up to four WASM files are generated with different hashes, but the files are all identical copies of my source WASM file.

Unfortunately, Webpack 5 is the only way to go with Next.js 12, and the importing module must remain dynamic, because of the weight of the WASM in question (and not be SSR'd due to the use of client-only Web APIs).

[TimoWilhelm]

How are you importing the wasm module? I use wasm-pack to generate the glue code and generate an ES module which I can then import in my app like so:

import dynamic from 'next/dynamic';

const Page = dynamic({
  loader: async () => {
    const module = await import('wasm-module');
    ...
  }
});

My full workaround for the moment:

/**
 * @param {boolean} isServer
 * @param {import("webpack").Configuration} config
 */
function patchWasmModuleImport(isServer, config) {
  config.experiments = Object.assign(config.experiments || {}, {
    asyncWebAssembly: true,
  });

  config.module.rules.push({
    test: /\.wasm$/,
    type: 'webassembly/async',
  });

  // TODO: cleanup -> track https://github.com/vercel/next.js/issues/25852
  if (isServer) {
    config.output.webassemblyModuleFilename =
      './../static/wasm/[modulehash].wasm';
  } else {
    config.output.webassemblyModuleFilename = 'static/wasm/[modulehash].wasm';
  }
}

[franky47]

My import tree is a bit deeper:

1. _app.tsx dynamically imports a worker.controller.ts file
2. worker.controller.ts creates a Web Worker: new Worker('path/to/worker/main')
3. path/to/worker/main loads the WASM via the wasm-pack generated interface JS module

Here's the issue:

The same WASM file is generated under four output files, two in .next/server/static/wasm and two in .next/static/wasm. The build fails when trying to find a file with a hash that is not generated in the right location.

[TimoWilhelm]

That seems strange. I'm also using the worker-url webpack plugin to load a worker and then dynamically importing a wasm-pack module there. But I have no idea why multiple modules are generated in your case.

[franky47]

I finally managed to get a hack of my own to work: I'm symlinking .next/server/static/wasm to point to .next/server/chunks/static/wasm, so that resolution can occur.

It requires cleanDistDir: false in the config to maintain the symlink, so it's a dirty hack, hopefully we can get this figured out soon.

[LeviticusNelson]

@franky47 can you show how you did this hack? I have the same issue that I would like to patch.

[franky47]

Here's the script I run before next build:

$  mkdir -p .next/server/chunks/static/wasm \
&& mkdir -p .next/server/static \
&& cd .next/server/static \
&& ln -s ../chunks/static/wasm wasm

And in next.config.js:

/** @type {import('next').NextConfig} */
const nextConfig = {
  cleanDistDir: false,
  webpack: config => {
    config.experiments.asyncWebAssembly = true
    return config
  }
}

[TimoWilhelm]

Nice idea! I made an webpack plugin to create the symlink, so it doesn't depend on cleanDistDir.

config.plugins.push(
  new (class {
    apply(compiler) {
      compiler.hooks.afterEmit.tapPromise(
        'SymlinkWebpackPlugin',
        async (compiler) => {
          if (isServer) {
            const from = join(compiler.options.output.path, '../static');
            const to = join(compiler.options.output.path, 'static');

            try {
              await access(from);
              console.log(`${from} already exists`);
              return;
            } catch (error) {
              if (error.code === 'ENOENT') {
                // No link exists
              } else {
                throw error;
              }
            }

            await symlink(to, from, 'junction');
            console.log(`created symlink ${from} -> ${to}`);
          }
        },
      );
    }
  })(),
);

[franky47]

Sweet! I was digging through the Webpack docs to figure out how to do exactly that, but gave up after an hour of digging.

Thanks for that, it works perfectly! ❤️

[LeviticusNelson]

Nice idea! I made an webpack plugin to create the symlink, so it doesn't depend on cleanDistDir.

config.plugins.push(
  new (class {
    apply(compiler) {
      compiler.hooks.afterEmit.tapPromise(
        'SymlinkWebpackPlugin',
        async (compiler) => {
          if (isServer) {
            const from = join(compiler.options.output.path, '../static');
            const to = join(compiler.options.output.path, 'static');

            try {
              await access(from);
              console.log(`${from} already exists`);
              return;
            } catch (error) {
              if (error.code === 'ENOENT') {
                // No link exists
              } else {
                throw error;
              }
            }

            await symlink(to, from, 'junction');
            console.log(`created symlink ${from} -> ${to}`);
          }
        },
      );
    }
  })(),
);

This works for me locally but I still get an error when I deploy on Vercel. This is the error I get from pages/api: {"errno":-2,"code":"ENOENT","syscall":"open","path":"/var/task/.next/server/static/wasm/ec38e8a176090534.wasm"}

[franky47]

Vercel being serverless might be peculiar about what Node.js APIs you can use (here node:fs/promises and node:path).

Usually there's a solution in configuring Vercel for runtime filesystem emulation, but since this is a build-time issue, it won't be of any use here.

[benmerckx]

This works for me locally but I still get an error when I deploy on Vercel. This is the error I get from pages/api: {"errno":-2,"code":"ENOENT","syscall":"open","path":"/var/task/.next/server/static/wasm/ec38e8a176090534.wasm"}

That seems to be https://github.com/vercel/next.js/issues/32612

[michalzalobny]

Nice idea! I made an webpack plugin to create the symlink, so it doesn't depend on cleanDistDir.

config.plugins.push(
  new (class {
    apply(compiler) {
      compiler.hooks.afterEmit.tapPromise(
        'SymlinkWebpackPlugin',
        async (compiler) => {
          if (isServer) {
            const from = join(compiler.options.output.path, '../static');
            const to = join(compiler.options.output.path, 'static');

            try {
              await access(from);
              console.log(`${from} already exists`);
              return;
            } catch (error) {
              if (error.code === 'ENOENT') {
                // No link exists
              } else {
                throw error;
              }
            }

            await symlink(to, from, 'junction');
            console.log(`created symlink ${from} -> ${to}`);
          }
        },
      );
    }
  })(),
);

When I try to push this plugin it breaks and says that It could not find the join, access, and symlink functions. Where are they declared in your webpack code?

[typeofweb]

@michalzalobny these are from node.js bult-in modules - join is from node:path while access and symlink are from node:fs/promises.

[iksent]

Didn't worked without config.optimization.moduleIds = 'named';.

So the result workaround for me is:

webpack: (config, options) => {
    patchWasmModuleImport(config, options.isServer);
    return config;
},

+

function patchWasmModuleImport(config, isServer) {
    config.experiments = Object.assign(config.experiments || {}, {
        asyncWebAssembly: true,
    });

    config.optimization.moduleIds = 'named';

    config.module.rules.push({
        test: /\.wasm$/,
        type: 'webassembly/async',
    });

    // TODO: improve this function -> track https://github.com/vercel/next.js/issues/25852
    if (isServer) {
        config.output.webassemblyModuleFilename = './../static/wasm/[modulehash].wasm';
    } else {
        config.output.webassemblyModuleFilename = 'static/wasm/[modulehash].wasm';
    }
}

[Yash3443]

`const CopyPlugin = require("copy-webpack-plugin"); /* @type {import('next').NextConfig} / const nextConfig = { webpack: function (config, options) { config.plugins.push(new CopyPlugin({ patterns: [ { from: "public/wasm", to: "./static/wasm" }, ], })) return config; } }

module.exports = nextConfig `

It resolved my issue related to including wasm file in bundle