search

rustwasm / wasm-pack

📦✨ your favorite rust -> wasm workflow tool!
https://rustwasm.github.io/wasm-pack/
Apache License 2.0
6.23k stars 406 forks source link

`wasm-pack` should generate isomorphic code #1334

Open loynoir opened 1 year ago

loynoir commented 1 year ago

💡 Feature description

wasm-pack should generate isomorphic code

💻 Basic example

Below two targets have different output.

$ cd hello-world
$ wasm-pack build -t nodejs
$ wasm-pack build -t web

Differences are

But,

So,

Related

cryptoquick commented 1 year ago

Please :pray:

currenthandle commented 11 months ago

Bump. This would be incredible to have. Is anyone on the team working on this? Can we support in anyway? 🫡

Denommus commented 11 months ago

Isomorphic code would be amazing.

loynoir commented 11 months ago

Workaround, tested same code works on both node and browser.

wasm-pack build --target nodejs --out-dir ./dist/pkg && node ./patch.mjs
import { readFile, writeFile } from "node:fs/promises";

const name = "xxx";

const content = await readFile(`./dist/pkg/${name}.js`, "utf8");

const patched = content
  // use global TextDecoder TextEncoder
  .replace("require(`util`)", "globalThis")
  // inline bytes Uint8Array
  .replace(
    /\nconst path.*\nconst bytes.*\n/,
    `
var __toBinary = /* @__PURE__ */ (() => {
  var table = new Uint8Array(128);
  for (var i = 0; i < 64; i++)
    table[i < 26 ? i + 65 : i < 52 ? i + 71 : i < 62 ? i - 4 : i * 4 - 205] = i;
  return (base64) => {
    var n = base64.length, bytes = new Uint8Array((n - (base64[n - 1] == "=") - (base64[n - 2] == "=")) * 3 / 4 | 0);
    for (var i2 = 0, j = 0; i2 < n; ) {
      var c0 = table[base64.charCodeAt(i2++)], c1 = table[base64.charCodeAt(i2++)];
      var c2 = table[base64.charCodeAt(i2++)], c3 = table[base64.charCodeAt(i2++)];
      bytes[j++] = c0 << 2 | c1 >> 4;
      bytes[j++] = c1 << 4 | c2 >> 2;
      bytes[j++] = c2 << 6 | c3;
    }
    return bytes;
  };
})();

const bytes = __toBinary(${JSON.stringify(await readFile(`./dist/pkg/${name}_bg.wasm`, "base64"))
    });
`,
  );

// deal with `imports['__wbindgen_placeholder__']`
// TODO: optimize with `__wbg_get_imports`
const wrapped = `export default (function() {
  const module = { exports: {} };

  ${patched}

  return module.exports;
})()
`;

await writeFile(`./dist/${name}.mjs`, wrapped);
microdou commented 11 months ago

@loynoir Awesome work! I love how you made it isomorphic. Previously, I was doing something similar, inlining base64 to bring wasm-pack libraries (web target) into a web worker within a SvelteKit project (because compiled SvelteKit web worker has difficulty in fetching wasm file unless inlined). I didn't think it could be extended this far, so kudos for that! 🤩

I have a quick question though. Your approach allows importing the entire wasm-pack lib like so:

import wasm from "my-wasm-lib";
const { add } = wasm;

console.log(add(1, 2))

Would it be feasible to import individual functions directly? Something akin to:

import { add } from "my-wasm-lib";

console.log(add(1, 2))
loynoir commented 11 months ago 
import { readFile, writeFile } from "node:fs/promises";

const cargoTomlContent = await readFile("./Cargo.toml", "utf8");
const cargoPackageName = /\[package\]\nname = "(.*?)"/.exec(cargoTomlContent)[1]
const name = cargoPackageName.replace(/-/g, '_')

const content = await readFile(`./dist/pkg/${name}.js`, "utf8");

const patched = content
  // use global TextDecoder TextEncoder
  .replace("require(`util`)", "globalThis")
  // attach to `imports` instead of module.exports
  .replace("= module.exports", "= imports")
  .replace(/\nmodule\.exports\.(.*?)\s+/g, "\nexport const $1 = imports.$1 ")
  .replace(/$/, 'export default imports')
  // inline bytes Uint8Array
  .replace(
    /\nconst path.*\nconst bytes.*\n/,
    `
var __toBinary = /* @__PURE__ */ (() => {
  var table = new Uint8Array(128);
  for (var i = 0; i < 64; i++)
    table[i < 26 ? i + 65 : i < 52 ? i + 71 : i < 62 ? i - 4 : i * 4 - 205] = i;
  return (base64) => {
    var n = base64.length, bytes = new Uint8Array((n - (base64[n - 1] == "=") - (base64[n - 2] == "=")) * 3 / 4 | 0);
    for (var i2 = 0, j = 0; i2 < n; ) {
      var c0 = table[base64.charCodeAt(i2++)], c1 = table[base64.charCodeAt(i2++)];
      var c2 = table[base64.charCodeAt(i2++)], c3 = table[base64.charCodeAt(i2++)];
      bytes[j++] = c0 << 2 | c1 >> 4;
      bytes[j++] = c1 << 4 | c2 >> 2;
      bytes[j++] = c2 << 6 | c3;
    }
    return bytes;
  };
})();

const bytes = __toBinary(${JSON.stringify(await readFile(`./dist/pkg/${name}_bg.wasm`, "base64"))
    });
`,
  );

await writeFile(`./dist/${name}.mjs`, patched);
microdou commented 11 months ago

@loynoir Works brilliantly! Thanks!

syczuan commented 10 months ago

@loynoir Thanks for the solution, I'm using nuxt.js and I can use it in both server and client environments with just a little code modification .replace("require('util')", "process.client?globalThis:require('util')")

lvauvillier commented 2 months ago

Thanks a lot for this script. I updated it to make it work for both exported functions and classes:

The following line is too generic:

.replace(/\nmodule\.exports\.(.*?)\s+/g, "\nexport const $1 = imports.$1 ")

It produces duplicated class identifier declarations:

class MyClass { }
//...
export const MyClass = imports.MyClass = MyClass;

I added these lines before to add the suffix "Class":

// to add before
.replace(/\nclass (.*?) \{/g, "\nclass $1Class {")
.replace(/\nmodule\.exports\.(.*?) = \1;/g, "\nexport const $1 = imports.$1 = $1Class")

It produces:

class MyClassClass { }
//...
export const MyClass = imports.MyClass = MyClassClass;
torfmaster commented 1 month ago

FYI: I created a draft PR here which might touch this issue: https://github.com/rustwasm/wasm-bindgen/pull/4065

ahaoboy commented 1 month ago

This is a very common use case, why is it still not supported from 2019 to now? swc implements a similar requirement using regular expressions https://github.com/swc-project/swc/blob/main/bindings/binding_typescript_wasm/scripts/patch.mjs

A command line tool to bundle the products of wasm-pack into a single js file https://github.com/ahaoboy/wasm-pack-inline

Related https://github.com/rustwasm/wasm-pack/issues/1074 https://github.com/rustwasm/wasm-pack/issues/831 https://github.com/rustwasm/wasm-pack/issues/699