Open gthb opened 3 years ago
The yarn run dev
problem is fixed in v11.1.3-canary.13, was still broken in v11.1.3-canary.7, so something inbetween there fixed it (maybe the webpack upgrade).
But yarn run build
still fails in the same way, in that canary.13 build. So I guess it's not actually the (exact) same problem.
In the latest canary build, v11.1.3-canary.32, yarn run build
now fails on something else, presumably unrelated:
yarn run v1.22.11
$ next build
warn - You have enabled experimental feature(s).
warn - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.
info - Checking validity of types
> Build error occurred
Error: 'entryOptions.layer' is only allowed when 'experiments.layers' is enabled
at Function.entryDescriptionToOptions (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:36878:10)
at Function.applyEntryOption (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:36846:39)
at /Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:36827:22
at Hook.eval [as call] (eval at create (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:139224:10), <anonymous>:7:16)
at Hook.CALL_DELEGATE [as _call] (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:139036:14)
at WebpackOptionsApply.process (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:60381:30)
at createCompiler (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:130634:28)
at create (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:130680:16)
at webpack (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:130704:32)
at Object.f [as webpack] (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:86594:16)
at /Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/build/compiler.js:31:40
at new Promise (<anonymous>)
at Object.runCompiler (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/build/compiler.js:30:12)
at /Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/build/index.js:358:59
at async Span.traceAsyncFn (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/trace/trace.js:73:20)
at async /Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/build/index.js:343:9
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
But enabling that layers
experiment makes this go away so we can test our problem ... and yep, yarn run build
still fails with the same Error: ENOENT: no such file or directory
problem as described above, in the latest canary build, canary.32.
As per #29485, I'm actually not sure this is wasm-pack
's (or wasm-bindgen
's fault). I'm seeing this behaviour with just importing the vanilla .wasm
file when building the Next.js application for production. Out of interest, are you able to replicate the problem in my issue?
I believe it is #22697 that's causing this conflict.
On a production build with webpack 5 the output configuration is as follows:
{
publicPath: '/_next/',
path: '/Users/sam/Desktop/with-webassembly-app/.next/server/chunks',
filename: '../[name].js',
library: undefined,
libraryTarget: 'commonjs2',
hotUpdateChunkFilename: 'static/webpack/[id].[fullhash].hot-update.js',
hotUpdateMainFilename: 'static/webpack/[fullhash].[runtime].hot-update.json',
chunkFilename: '[name].js',
strictModuleExceptionHandling: true,
crossOriginLoading: undefined,
webassemblyModuleFilename: 'static/wasm/[modulehash].wasm'
}
It seems as though webassemblyModuleFilename
interprets this to output to .next/server/chunks/static/wasm/[modulehash].wasm
, or alternatively {output.path}/{webassemblyModuleFilename}
. Chunks are outputted to .next/server/chunks
but ordinary files are outputted to the parent directory with ../[name].js
. webassemblyModuleFilename
doesn't handle this parent recursion correctly, and so ends up in chunks
. When importing, it doesn't appear to look for chunks
because it's looking for webassemblyModuleFilename
at the root of the webpack config, or .next/server
@gthb Lord forgive me, but I've created a workaround:
// next.config.js
module.exports = {
webpack(config, { isServer, dev }) {
// Enable webassembly
config.experiments = { asyncWebAssembly: true };
// In prod mode and in the server bundle (the place where this "chunks" bug
// appears), use the client static directory for the same .wasm bundle
config.output.webassemblyModuleFilename =
isServer && !dev ? "../static/wasm/[id].wasm" : "static/wasm/[id].wasm";
// Ensure the filename for the .wasm bundle is the same on both the client
// and the server (as in any other mode the ID's won't match)
config.optimization.moduleIds = "named";
return config;
},
};
Then to ensure the .wasm
file is always included in the client bundle (you don't need to do this if you're using it ONLY on the client, or in both the server and the client), place this somewhere in the root of your _app.tsx
:
(function () {
import("wasm/add.wasm");
// Import which .wasm files you need here
});
This won't download the file onto the client, just ensure that it's in the bundle
This is obviously far from an ideal solution, because it still generates the server bundle at .next/server/chunks/../static/wasm/[id].wasm
(i.e. .next/server/static/wasm/[id].wasm
) that webpack is trying to import for the server, but instead gets the static
one because it's looking for .next/server/../static/wasm/[id].wasm
(i.e. .next/static/wasm/[id].wasm
)
The above workaround breaks when using experiments.layers = true
because even though the module ID is named, a chunk hash is still appended to the output assets. The only way the above fix can work is if the client and server output the same filename for both the client and the sever.
I've created a more reliable workaround in the form of an embedded webpack plugin. This behaviour is pretty close to how Next.js internally handles the chunks
directory and then walking back up with ../
// next.config.js
module.exports = {
webpack(config, { isServer, dev }) {
config.experiments = {
asyncWebAssembly: true,
layers: true,
};
if (!dev && isServer) {
config.output.webassemblyModuleFilename = "chunks/[id].wasm";
config.plugins.push(new WasmChunksFixPlugin());
}
return config;
},
};
class WasmChunksFixPlugin {
apply(compiler) {
compiler.hooks.thisCompilation.tap("WasmChunksFixPlugin", (compilation) => {
compilation.hooks.processAssets.tap(
{ name: "WasmChunksFixPlugin" },
(assets) =>
Object.entries(assets).forEach(([pathname, source]) => {
if (!pathname.match(/\.wasm$/)) return;
compilation.deleteAsset(pathname);
const name = pathname.split("/")[1];
const info = compilation.assetsInfo.get(pathname);
compilation.emitAsset(name, source, info);
})
);
});
}
}
The above workaround breaks when using
experiments.layers = true
because even though the module ID is named, a chunk hash is still appended to the output assets. The only way the above fix can work is if the client and server output the same filename for both the client and the sever.I've created a more reliable workaround in the form of an embedded webpack plugin. This behaviour is pretty close to how Next.js internally handles the
chunks
directory and then walking back up with../
// next.config.js module.exports = { webpack(config, { isServer, dev }) { config.experiments = { asyncWebAssembly: true, layers: true, }; if (!dev && isServer) { config.output.webassemblyModuleFilename = "chunks/[id].wasm"; config.plugins.push(new WasmChunksFixPlugin()); } config.module.rules.push({ test: /\.svg$/, use: ["@svgr/webpack"], }); return config; }, }; class WasmChunksFixPlugin { apply(compiler) { compiler.hooks.thisCompilation.tap("WasmChunksFixPlugin", (compilation) => { compilation.hooks.processAssets.tap( { name: "WasmChunksFixPlugin" }, (assets) => Object.entries(assets).forEach(([pathname, source]) => { if (!pathname.match(/\.wasm$/)) return; compilation.deleteAsset(pathname); const name = pathname.split("/")[1]; const info = compilation.assetsInfo.get(pathname); compilation.emitAsset(name, source, info); }) ); }); } }
Oh man! 4 hours later I bump into this thread.
I had packed a battle snake solver with wasm-pack
and had placed it on my pages/api
routes, everything worked fine locally, but then the madness began when trying to publish it.
However this seems to work only with version 11, on Next 12, it breaks during import, I guess I gotta read further.
Thanks for the snippet! @sam3d !
@icyJoseph is it breaking when you try and deploy it on Vercel? The snippet is working for me in Next 12 for both dev and prod build locally. However on Vercel it seems like output file tracing doesn't detect the .wasm file. I believe the place this is happening in @vercel/nft is here:
@icyJoseph is it breaking when you try and deploy it on Vercel? The snippet is working for me in Next 12 for both dev and prod build locally. However on Vercel it seems like output file tracing doesn't detect the .wasm file. I believe the place this is happening in @vercel/nft is here:
Ah I spoke too soon, when I deployed it on vercel it fails on, with Next 11.
ERROR TypeError: wasm.get_info is not a function
And locally with next 12, it all falls apart.
Here's how I import the wasm file, on pages/piage/index.js
export default async function handler(req, res) {
if (req.method !== "GET") return res.status(404).json({ error: "No" });
try {
const wasm = await import("../../../snake/pkg/logic");
return res.status(200).json(wasm.get_info());
} catch (e) {
console.error(e);
return res.status(500).json({ message: "Internal Server Error" });
}
}
Here's a link to my config https://github.com/icyJoseph/vercel-battle-snake-rust/blob/fix-runtime/next.config.js, on the branch where I am trying to get this sorted out.
I think for now I'll just solve these with TypeScript.
@icyJoseph What happens if you remove the WasmPackPlugin? I don't think they're interoperable
@sam3d I had to take a moment to re-do the entire thing, my code had changed quite a lot from the fork. Now I have it all running on TypeScript, and a sub route /api/wasm
where I am trying to get one function to work correctly.
https://github.com/icyJoseph/next-battlesnake/tree/with-wasm -> set at the correct branch
I did as you suggested, and now things work fine locally, with Next 12, but still, when deploying to Vercel, upon reaching for the /api/wasm
endpoint I get:
2021-11-23T12:54:00.429Z acc7efcd-6066-45df-8452-37d1feb32982 ERROR [Error: ENOENT: no such file or directory, open '/var/task/.next/server/chunks/817.wasm'] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/var/task/.next/server/chunks/817.wasm'
}
Looking at https://github.com/vercel/nft/blob/e89f3ca8cfa41838d0c4d7c16255eba7f561609f/src/node-file-trace.ts#L40-L42, as you suggest, seems very telling, have you had a chance to try and debug it or make PR?
if (path.endsWith('.js') || path.endsWith('.cjs') || path.endsWith('.mjs') || path.endsWith('.node') || job.ts && (path.endsWith('.ts') || path.endsWith('.tsx'))) {
return job.emitDependency(path);
}
@icyJoseph I filed this issue about a week ago (https://github.com/vercel/nft/issues/247) but I'm afraid I haven't yet the chance to work on a reproduction or PR for it (as I don't have the Next.js dev stack configured locally). My assumption is that a simple addition of path.endsWith('.wasm')
would fix the issue, but I don't know that for sure
(p.s. you don't need the following by the way, I posted my patch hastily and forgot to remove this part from my own Next.js config:)
config.module.rules.push({
test: /\.svg$/,
use: ["@svgr/webpack"],
});
@icyJoseph (yet another) temporary fix could be something like this: https://github.com/vercel/next.js/discussions/14807 except specifying the chunks directory with .wasm
files, though I haven't been able to test this either
@sam3d Ah I gave it a go, but I am not sure how am I to point to the wasm file, if its hashed, and placed inside .next, because that's the one that can't be found. I guess I'll skip the WASM version of this for now.
Hi has there been any progress on a fix for the deployment on Vercel? I have the same exact issue as @icyJoseph
This worked for me. So my next-config.js file was.
` /* @type {import('next').NextConfig} / const nextConfig = { reactStrictMode: true, }
module.exports = nextConfig
module.exports = { webpack: (config, { isServer }) => { // it makes a WebAssembly modules async modules config.experiments = { syncWebAssembly: true, asyncWebAssembly: true, layers: true }
// generate wasm module in ".next/server" for ssr & ssg if (isServer) { config.output.webassemblyModuleFilename = './../static/wasm/[modulehash].wasm' } else { config.output.webassemblyModuleFilename = 'static/wasm/[modulehash].wasm' }
return config }, } `
π¨ This comment is partially out of date β see below
I managed to get import * as wasm from "./my.wasm"
working during next dev
/ next build
/ next start
, thanks to a workaround by @ctrespoo.
Here is my next.config.js
for next@12.3.2-canary.27
(UPD: and next@13.0.2
):
/** @type {import("next").NextConfig} */
export default {
webpack: (webpackConfig, { isServer }) => {
// WASM imports are not supported by default. Workaround inspired by:
// https://github.com/vercel/next.js/issues/29362#issuecomment-1149903338
// https://github.com/vercel/next.js/issues/32612#issuecomment-1082704675
return {
...webpackConfig,
experiments: {
asyncWebAssembly: true,
layers: true,
},
optimization: {
...webpackConfig.optimization,
moduleIds: "named",
},
output: {
...webpackConfig.output,
webassemblyModuleFilename: isServer
? "./../static/wasm/[modulehash].wasm"
: "static/wasm/[modulehash].wasm",
},
};
},
};
WASM modules load fine in an SSR page, but an API route returns Internal Server Error
. Server logs contain this:
[Error: ENOENT: no such file or directory, open '/path/to/project/.next/static/wasm/7137689891e3e4e4.wasm'] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/path/to/project/.next/static/wasm/7137689891e3e4e4.wasm'
}
Both Next page and API handler works correctly in next dev
.
Repo with reproduction: hasharchives/wasm-ts-esm-in-node-jest-and-nextjs
Having similar issues as above ^. Anyone any idea?
What happens when you use the workaround in https://github.com/vercel/next.js/issues/29362#issuecomment-971377869?
Oh wow thanks for the pointer @sam3d β I have missed that comment somehow. I just tried your workaround in hasharchives/wasm-ts-esm-in-node-jest-and-nextjs and it did itβs job! π Awesome stuff!
I was able to get a WASM module compiled from Rust with wasm-pack working both locally and on Vercel, except in a non-edge API route on Vercel.
Location | WASM in API Route? | WASM in Edge API Route? | WASM in Edge Middleware? |
---|---|---|---|
Local | β | β | β |
Vercel | β | β | β |
When deployed to Vercel, the logs show the following error when trying to call a WASM module from a non-edge API route:
[Error: ENOENT: no such file or directory, open '/var/task/apps/example-next/.next/server/static/wasm/8703cf278d4beedc.wasm'] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/var/task/apps/example-next/.next/server/static/wasm/8703cf278d4beedc.wasm'
}
[Error: ENOENT: no such file or directory, open '/var/task/apps/example-next/.next/server/static/wasm/8703cf278d4beedc.wasm'] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/var/task/apps/example-next/.next/server/static/wasm/8703cf278d4beedc.wasm'
}
RequestId: X Error: Runtime exited with error: exit status 1
Runtime.ExitError
I'm assuming this is expected because the docs state that WebAssembly is only supported in Edge Functions and Edge Middleware. But has anyone been able to get WASM working on a non-edge route on Vercel?
The example at https://github.com/hasharchives/wasm-ts-esm-in-node-jest-and-nextjs/blob/main/web-app/pages/api/wasm-package-answer.ts looked like it was successfully calling a WASM module from a non-edge API route, but perhaps that was only working locally.
I am using Nextjs13 with wasm. This thread has been a real help. π€
Though I would prefer a more official fix.
I'm facing a similar issue trying to use node-webpmux, which is compiled with Emscripten, and I'm so glad I found this thread. Unfortunately the workaround above by @sam3d is not working for me.
For context, I am trying to use this library in an API route, but I get the following runtime error:
Aborted(Error: ENOENT: no such file or directory, open '/Users/jordan/Documents/Coding/LED Matrix Display/next-renderer/.next/server/app/api/render/nyctrainsign/libwebp.wasm')
failed to asynchronously prepare wasm: RuntimeError: Aborted(Error: ENOENT: no such file or directory, open '/Users/jordan/Documents/Coding/LED Matrix Display/next-renderer/.next/server/app/api/render/nyctrainsign/libwebp.wasm'). Build with -s ASSERTIONS=1 for more info.
Aborted(RuntimeError: Aborted(Error: ENOENT: no such file or directory, open '/Users/jordan/Documents/Coding/LED Matrix Display/next-renderer/.next/server/app/api/render/nyctrainsign/libwebp.wasm'). Build with -s ASSERTIONS=1 for more info.)
- error RuntimeError: Aborted(Error: ENOENT: no such file or directory, open '/Users/jordan/Documents/Coding/LED Matrix Display/next-renderer/.next/server/app/api/render/nyctrainsign/libwebp.wasm'). Build with -s ASSERTIONS=1 for more info.
at abort (webpack-internal:///(rsc)/./node_modules/node-webpmux/libwebp/libwebp.js:495:21)
at getBinary (webpack-internal:///(rsc)/./node_modules/node-webpmux/libwebp/libwebp.js:522:17)
at eval (webpack-internal:///(rsc)/./node_modules/node-webpmux/libwebp/libwebp.js:549:24)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
I see that the library is trying to load libwebp.wasm
which exists in node_modules
but doesn't appear to be packed correctly into the .next
directory. Is this related to the above? Has anyone encountered something similar?
If it's helpful, I'm using NextJS 13.4.19
and node-webpmux 3.1.8
on Apple Silicon.
I was able to get a WASM module compiled from Rust with wasm-pack working both locally and on Vercel, except in a non-edge API route on Vercel.
Location WASM in API Route? WASM in Edge API Route? WASM in Edge Middleware? Local β β β Vercel β β β When deployed to Vercel, the logs show the following error when trying to call a WASM module from a non-edge API route:
[Error: ENOENT: no such file or directory, open '/var/task/apps/example-next/.next/server/static/wasm/8703cf278d4beedc.wasm'] { errno: -2, code: 'ENOENT', syscall: 'open', path: '/var/task/apps/example-next/.next/server/static/wasm/8703cf278d4beedc.wasm' } [Error: ENOENT: no such file or directory, open '/var/task/apps/example-next/.next/server/static/wasm/8703cf278d4beedc.wasm'] { errno: -2, code: 'ENOENT', syscall: 'open', path: '/var/task/apps/example-next/.next/server/static/wasm/8703cf278d4beedc.wasm' } RequestId: X Error: Runtime exited with error: exit status 1 Runtime.ExitError
I'm assuming this is expected because the docs state that WebAssembly is only supported in Edge Functions and Edge Middleware. But has anyone been able to get WASM working on a non-edge route on Vercel?
The example at https://github.com/hasharchives/wasm-ts-esm-in-node-jest-and-nextjs/blob/main/web-app/pages/api/wasm-package-answer.ts looked like it was successfully calling a WASM module from a non-edge API route, but perhaps that was only working locally.
I'm running into the same issue but with both Next App API routes and Server Actions. When API routes or Server Actions import WASM, it builds fine using the fix above, but when deployed to Vercel the WASM file doesn't exist and leads to "ENOENT: no such file or directory" upon server code execution.
@calclavia I was able to get WASM imports to work for regular API routes and on the frontend with the config below. Note that I had to enable Node 20 in the runtime settings (in Node 20 WASM support no longer requires a flag).
const CopyPlugin = require("copy-webpack-plugin");
const nextConfig = {
webpack: (config, { isServer, dev }) => {
config.experiments = {
layers: true,
asyncWebAssembly: true
};
if (!dev && isServer) {
webassemblyModuleFilename = "./../server/chunks/[modulehash].wasm";
const patterns = [];
const destinations = [
"../static/wasm/[name][ext]", // -> .next/static/wasm
"./static/wasm/[name][ext]", // -> .next/server/static/wasm
"." // -> .next/server/chunks (for some reason this is necessary)
];
for (const dest of destinations) {
patterns.push({
context: ".next/server/chunks",
from: ".",
to: dest,
filter: (resourcePath) => resourcePath.endsWith(".wasm"),
noErrorOnMissing: true
});
}
config.plugins.push(new CopyPlugin({ patterns }));
}
return config;
}
};
What version of Next.js are you using?
11.1.2
What version of Node.js are you using?
14.17.6
What browser are you using?
Firefox, Chrome
What operating system are you using?
macOS
How are you deploying your application?
Not yet deployed
Describe the Bug
A simple Rust WebAssembly module packaged with its glue code into an ES module with wasm-pack (patched as in https://github.com/rustwasm/wasm-pack/pull/1061) loads and works just fine under webpack, as illustrated in https://github.com/webpack/webpack/pull/14313, but fails to import under Next.js. This is apparently because Next.js generates the
.wasm
generated at one path but then tries to load it from a different path.Expected Behavior
I expected the
npm run dev
to successfully run the application and render a page with the greeting from inside the WebAssembly module.I expected
npm run build
to successfully build a production distribution that would do the same.To Reproduce
This is demonstrated in https://github.com/gthb/try-to-use-wasm-in-next.js/ β the README there recounts my circuitous path of trying to get this to work, but the current state serves to illustrate the problem I'm reporting here.
In that repo, first setup:
Then try
yarn run dev
β it fails like this:Then try
yarn run build
β it fails like this:Note the path to the
.wasm
chunk. The actual generated.wasm
chunks are:So apparently the failure is that Next.js generates the
.wasm
chunk on the server side with an extrachunks/
path element (which presumably should not be there) and then looks for it at a path without that path element and fails to find it.I'm guessing that the same problem is the cause of the
npm run dev
failure (i.e. the module imports as empty).