Sharing the same WebAssembly file across multiple targets

[magic-akari]

Summary

Compiling wasm for multiple targets is a common scenario. Is it possible to share the same wasm and only have different JavaScript glue code?

Additional Details

related issue:

• JohnnyMorganz/StyLua#848

[daxpedda]

I honestly don't know as I only use the Web target, but I don't remember any Wasm changes related to the target being made by wasm-bindgen-cli.

A very simple way to find out is just give it a try, if the Wasm file produced by wasm-bindgen-cli is byte-equivalent between different targets it should be fine.

[Swolebrain]

I looked at implementing this but since most people use wasm-bindgen through wasm-pack, it looked like it would be a huge rippling change.

So I opted to hack around it by creating a build script that builds all 3 targets, moves them into subdirectories of my pkg folder, and then arranges for the root package.json to have subpackages. You can take a look at this for reference https://github.com/cedar-policy/cedar/blob/main/cedar-wasm/build-wasm.sh

[magic-akari]

This is an attempt to share a wasm file across multiple compilation targets.

wasm-bindgen-proposal.tar.gz

1. Project Structure

Under the src directory, lib.rs is the source code, and the file content is quite simple.

use wasm_bindgen::prelude::*;
#[wasm_bindgen]
pub fn err() -> Result<(), u8> {
    Err(1)
}

This is the most basic example I've found where the WASM imports include an additional function, which we need to address next.

2. Build

Executing build.sh will generate the wasm file, and the content of build.sh is as follows.

# web
wasm-pack build --target=web --out-dir=web
wasm2wat web/wasm_bindgen_proposal_bg.wasm -o web/wasm_bindgen_proposal_bg.wat

# nodejs
wasm-pack build --target=nodejs --out-dir=nodejs
wasm2wat nodejs/wasm_bindgen_proposal_bg.wasm -o nodejs/wasm_bindgen_proposal_bg.wat

# bundler
wasm-pack build --target=bundler --out-dir=bundler
wasm2wat bundler/wasm_bindgen_proposal_bg.wasm -o bundler/wasm_bindgen_proposal_bg.wat

This will generate three directories, namely web, nodejs, and bundler. After each build, we use the wasm2wat tool to convert the wasm file into a wat file for easier inspection.

3. Analysis

It can be observed that the content of the wasm files in the three generated directories is almost the same, but there is a slight difference. It lies in the wasm imports:

• The web compilation target uses wbg as the entry point

(import "wbg" "__wbindgen_number_new" (func (;0;) (type 6)))

• The nodejs compilation target uses __wbindgen_placeholder__ as the entry point

(import "__wbindgen_placeholder__" "__wbindgen_number_new" (func (;0;) (type 6)))

• The bundler compilation target uses ./wasm_bindgen_proposal_bg.js as the entry point

(import "./wasm_bindgen_proposal_bg.js" "__wbindgen_number_new" (func (;0;) (type 6)))

This only difference makes it difficult to share the wasm file. I tried to unify the entry in https://github.com/JohnnyMorganz/StyLua/pull/848 with some hack methods, but a unified entry from wasm-bindgen during generation would be more convenient.

4. Attempt

I duplicated the web directory to create a proposal directory and replaced wbg in the wat file with ./wasm_bindgen_proposal.js. This change is inconsequential for web and Node.js but guides the bundler to find the correct file to import.

We then convert the modified WAT file back to WASM using wat2wasm:

wat2wasm proposal/web_modified.wat -o proposal/wasm_bindgen_proposal.wasm

Next, we need to modify the glue files to make them work correctly in web, nodejs, and bundler environments.

First, we need to modify the aforementioned entry file

- function __wbg_get_imports() {
-     const imports = {};
-     imports.wbg = {};
-     imports.wbg.__wbindgen_number_new = function(arg0) {
-         const ret = arg0;
-         return addHeapObject(ret);
-     };
- 
-     return imports;
- }
+ export function __wbindgen_number_new(arg0) {
+     const ret = arg0;
+     return addHeapObject(ret);
+ };
+ 
+ function __wbg_get_imports() {
+     const imports = {};
+     imports["./wasm_bindgen_proposal.js"] = {
+         __wbindgen_number_new,
+     };
+ 
+     return imports;
+ }

We no longer use the wbg entry, but use the "./wasm_bindgen_proposal.js" entry, because we have already modified the imports in the wat file. At the same time, we need to export the __wbindgen_number_new function for the bundler to use.

Observing the bundler/wasm_bindgen_proposal.js file, we copy it to proposal/wasm_bindgen_proposal_bundle.js and modify its content as follows,

import * as wasm from "./wasm_bindgen_proposal_bg.wasm";
import { __wbg_set_wasm } from "./wasm_bindgen_proposal.js";
__wbg_set_wasm(wasm);
export * from "./wasm_bindgen_proposal.js";

So, we need to add and export the __wbg_set_wasm function in wasm_bindgen_proposal.js.

So far, the glue files for esm and bundle have been modified. Let's look at the glue file for the nodejs environment next. We copy the nodejs/wasm_bindgen_proposal.js file to proposal/wasm_bindgen_proposal.cjs and modify its content as follows,

- imports['__wbindgen_placeholder__'] = module.exports;
+ imports['./wasm_bindgen_proposal.js'] = module.exports;

In this way, we have completed the modification of the glue file for the nodejs environment.

5. Testing

In fact, what we generate are esm, cjs, and bundler, three compilation targets, which share the same wasm file, where esm can be executed in the browser, nodejs, deno, bun environments; cjs is used for the traditional nodejs environment; bundler is used for packaging tools like webpack.

We now write test files, which are test.mjs and test.cjs

import { initSync, err } from "./proposal/wasm_bindgen_proposal.js"
import fs from "node:fs";

const wasm = fs.readFileSync("./proposal/wasm_bindgen_proposal_bg.wasm");

initSync(wasm);

try {
    err();
} catch (error) {
    console.log(error);
}

const m = require("./proposal/wasm_bindgen_proposal.cjs");

try {
    m.err();
} catch (error) {
    console.log(error);
}

Because nodejs does not support the fetch file: protocol address, we need to use the fs module to read the wasm file. If testing in deno, you can directly use the async init method.

In the end, we see that the output results are all 1, which means our shared wasm file is successful.

Next, we test webpack compatibility, and the webpack configuration file is webpack.config.js Executing npm run build will call webpack to package, and it will output the webpack/node and webpack/web two directories, which are the packaging results for the nodejs and web environments, respectively.

We execute node webpack/node/main.js and can see that the output result is 1, indicating that our shared wasm file is also successful in the webpack environment. Create an index.html file and introduce the packaging results of the web environment, and we can see that the browser console output result is 1, indicating that our shared wasm file is also successful in the browser environment. (You may need an HTTP server to run the index.html file, such as python -m http.server or npx http-server or miniserve .)

6. Summary

Sharing WASM files is feasible by standardizing the import entry and adjusting glue code for different environments. This approach allows the same WASM file to be used across web, Node.js, and bundler compilation targets.

[magic-akari]

This is a proof of concept, demonstrating the potential for sharing Wasm file. I am curious if wasm-bindgen has plans to implement this feature.

[daxpedda]

This is a proof of concept, demonstrating the potential for sharing Wasm file. I am curious if wasm-bindgen has plans to implement this feature.

There are no plans to implement a feature like this. Though I'm happy to review a PR, I will only be able to review the Web part.