Compiling Rhai to WASM

[alvinhochun]

I'm asking specifically for the wasm32-unknown-unknown target on web browser using wasm-pack. It looks like there should be nothing preventing Rhai from working, but since it is not explicitly stated anywhere I think I should ask.

[schungx]

I happen to remember an post from reddit: https://www.reddit.com/r/rust/comments/h0gmph/embeddable_lualike_scripting_for_both_native_and/?utm_source=amp&utm_medium=&utm_content=post_body

Asking about the same thing. I sent the author a message asking whether he successfully got it working. I myself don't have the time and need to property test it out...

According to the post, it builds successfully.

However, may I ask why you want it in WASM? There is already a nice scripting language for WASM environments, and it is called JavaScript...

[alvinhochun]

Cool, I'll make a dirty demo to give it a try. What worries me is that some of Rust's libstd are stubbed with panics and I don't know if Rhai calls any of them.

I am making a game-like (but not actually a game but that's not important) application that needs custom functions (supplied in custom data files) and I thought that a small interpreted scripting language would be a good choice. My primary target is native (Windows + Linux) but I'd also like the ability to build a Wasm demo version without too many changes. I just thought I'd try Rhai because it's written in Rust :P

[alvinhochun]

I've hacked together a demo that can run scripts and log output: https://github.com/alvinhochun/wasm-project-template/tree/rhai-demo

[schungx]

What worries me is that some of Rust's libstd are stubbed with panics and I don't know if Rhai calls any of them.

There shouldn't be many. I thought I replaced all of them with unreachable!(). They are code paths not supposed to be hit.

I've hacked together a demo

Do you have a URL that we can test it? I simply can't wait..

[alvinhochun]

Do you have a URL that we can test it? I simply can't wait..

This should work: https://alvinhochun.github.io/rhai-demo/index.html

[schungx]

It says EXCEPTION: "ReferenceError: run_script is not defined"

[alvinhochun]

Weird, it works for me on both Firefox and Chrome...

[schungx]

OK. Got it. It hasn't finished downloading yet. Once it finishes, it works fine!

[schungx]

The WASM package is 800K gzipped though... not exactly light-weight... Maybe you try compiling it for size?

[alvinhochun]

It was a debug build... I've updated it with a release build and opt-level=s.

[schungx]

I am wondering, what if you turn on the following features: only_i32 or only_i64, no_module, no_optimize, unchecked and compile for size?

[schungx]

It was a debug build... I've updated it with a release build and opt-level=s.

Not bad! It is 187KB gzipped! I wonder can I put WASM support on the docs? :-D

[alvinhochun]

Now it is 158.22KB gzipped with features = [ "only_i64", "no_module", "no_optimize", "unchecked" ].

Is there like a script that, like, test every built-in functions and packages?

[schungx]

Now it is 158.22KB gzipped

Not as much savings as I'd like...

Is there like a script that, like, test every built-in functions and packages?

Hhhhhmmmm... not really...

I use the following script myself, but don't laugh:

print(type_of(123));
//let foo = 123.0;
//foo.y[2].x = 123;
let a = [#{}];
a[0].s = "Stephen";
print(a[0].s.len);
a[0].s[2] = 'X';
print(a[0].s);
a[0].list = [1, 3, 5, "hello", 9];
print(a[0].list[3][2]);
//return a[0];

/* Fear not, Rhai satisfies all your nesting
   needs with nested comments:
   /*/*/*/*/**/*/*/*/*/
   *********
   ///////////
*/

print("Here we start...");

print("### Empty array: " + [].len);

print("### Block as Expression:");

let x = {{{["hello"]}}};
print("block expr [hello]: " + x + ", len: " + {let z = x.len; z});

print("### Indexing:");

let x = [1,2,3,4,5][2];//[9][0];    // Remove // to get error
print("3 == " + x + ", type: " + type_of(x));

fn abc() { [42, 43, 44 ] }
print ("43 == " + abc()[1]);

print("4 == " + "12345"[3]);

//print("### Decide:" + decide(true) + ", " + decide(false));

print("### Conversions: ");

if 5 == "5" {
    throw "WRONG TYPE!";
}

let xxx = 0b01_101;
print(xxx);
print(xxx.to_float() + 6.0);
let xxx = -12345.6789;
print(to_int(xxx));
let yyy = 'Q';
print(yyy + ": code " + yyy.to_int());

print("### String manipulations: ");
let xxx = " hello   ";
xxx.trim();
xxx.replace("ll", "__");
print("Should be '_': " + xxx[3]);
print("Contains __: " + xxx.contains("__"));
xxx[3] = 'X';
xxx[4] = '&';
print("Changed string ('he__&'): " + xxx);

print("### Object: ");
let ts = #{s:"kitty", x:1, f:123.456, list:[]};
print(type_of(ts));

print("string prop: " + ts.s);
print("s[2]:"+ts.s[2]+", type:"+type_of(ts.s[2]));
let s = ts.s;
ts.s[3] = '@';
print("kit@y == " + ts.s);

print("### Define change");
fn change(s) {
    let xxx = 42;

    print("changing..." + s);
    s[0] = '#';
    print("changed: " + s);
}
ts.s.change();
print("outside: " + ts.s);
print("should not be 42, should be 'he__&': " + xxx);

if xxx == 42 {
    throw "SERIOUS ERROR!!!";
} else {
    print("passed!");
}

let y = [1, 2, 3, 4, "hello"];

print("### Original Array: " + y);
print("element y[2]: " + y[2]);

y.push(5);
print("pushed: " + y);

let z = y.shift();
debug("shifted: " + z);
print("result: " + y);

print("### Elements:");
for x in y {
    print(x);
}

ts.list = y;
ts.list.push('$');
print("ts.list: " + ts.list);
ts.x = ts.list.len;

print("ts.list[2]: " + ts.list[2]);
print("length: " + ts.x);

ts.f *= 100.0;
ts.x += 420000;

print("Type of ts: " + type_of(ts));

ts

[alvinhochun]

I found that at least timestamp() panics as std::time is not supported in Wasm.

[schungx]

I updated the script without depending on custom types. Now it runs fine on your URL!

[schungx]

I found that at least timestamp() panics as std::time is not supported in Wasm.

Maybe I'll need to put a target guard on it...

[schungx]

Do you think I need to guard against the file I/O functions as well? I see that the build compiles OK but there are file I/O calls in the code.

[alvinhochun]

File I/O are definitely no good for Wasm. I think we will need to go through all the functions and packages and list out what doesn't work on Wasm and whether there are alternatives (for example https://github.com/sebcrozet/instant can be a replacement for std::time::Instant).

I think these (1 2) searches (but better just ripgrep them offline) should list all the unsupported libstd functions, not sure if there is an official list (couldn't find one anyway). Then perhaps we can search through Rhai's source code for them? I wish there is a better way to do this.

[schungx]

I can go thru and gate them, it is not difficult. I simply search for all the places with no_std and that should about find them all.

However, I am wondering why you don't get a compilation error when compiling to WASM. I'm wondering if all the unsupported function calls simply panic in WASM instead of yield compile errors...

[schungx]

OK, I think I know why. As long as you don't use the file API's, those functions are simply eliminated by the optimizer and they don't get into the WASM. Still, it is probably a good idea to gate them so a user won't unknowingly use them.

[schungx]

@alvinhochun I have tried a couple of scripts (under scripts directory) on the WASM engine and they all work fine (I had to take away timestamp()).

Judging from my own stop-watch, the speed is a bit slower than an optimized native build on the command line, but not by much! At least not an order of magnitude. WASM is great!

[alvinhochun]

I am wondering why you don't get a compilation error when compiling to WASM. I'm wondering if all the unsupported function calls simply panic in WASM instead of yield compile errors...

OK, I think I know why. As long as you don't use the file API's, those functions are simply eliminated by the optimizer and they don't get into the WASM. Still, it is probably a good idea to gate them so a user won't unknowingly use them.

Yes, they just panic (though it is possible that some may just return Err instead, haven't checked). I can try to explain a bit with my very limited understanding on the Wasm situation...

There are actually three different Wasm targets (not counting asmjs) that one can use:

wasm32-unknown-emscripten

Emscripten is a pipeline that includes a libc implementation. It even includes a virtual filesystem to allow file I/O via libc. I believe that when it comes to Rust, this target is considered legacy.

wasm32-unknown-unknown

This target doesn't have a libc or any system interfaces really. It does not assume the host (hence the unknown-unknown) so one can use the Wasm modules outside of web browsers. The result is that the Rust standard library cannot rely on any of the JavaScript or Web APIs. They decided to not include everything like Emscripten do. But instead of leaving the target no_std only, they implemented whatever they could in std and leave the unimplemented ones to panic, so using them won't prevent compilation.

Some of the unimplemented stuff can actually be implemented on web browsers using the JavaScript or Web APIs, but in this target they require the application code to be aware of them and provide custom implementations. The std::time::Instant replacement I mentioned earlier is an example of this.

Web APIs can't be used directly from Wasm; to use them you need to call into JavaScript. To call into JavaScript, you need to use FFI. Fortunately there are crates that make things a lot easier.

wasm32-wasi

This is the latest new thing I believe. Wasi is a system interface designed for use by Wasm. It should mean that a lot more of Rust's std can be properly implemented (there are probably some exceptions). But web browsers currently doesn't support Wasi, so you will end up using polyfills (which would provide the interfaces through Web APIs) if you want to run the built Wasm modules on a web browser.

---

At this moment I'm really only focusing on targeting web browsers using wasm32-unknown-unknown with wasm-bindgen. Even though I won't need a lot of the built-in functions in Rhai, I would prefer to see them (e.g. timestamp()) implemented using JavaScript or Web APIs if needed, instead of removed by a cfg check. This should however be gated behind a feature flag (perhaps wasm_web?) so others may use Rhai with Wasm outside of web browsers if they wish to.

[alvinhochun]

Judging from my own stop-watch, the speed is a bit slower than an optimized native build on the command line, but not by much! At least not an order of magnitude. WASM is great!

The Wasm build I have there is also optimized for size instead of speed.

[schungx]

The Wasm build I have there is also optimized for size instead of speed.

Hhhmmm.... I wonder what difference it would be to compile it for speed instead... How much more code size vs how much higher speed...

would prefer to see them (e.g. timestamp()) implemented

Yes, I'll probably pull in an alternative crate for Instant if the wasm target is detected. For other things, I looked thru the code base, and it seems like the only thing that should panic (other than Instant) should be file I/O APIs...

I'm currently gating all the file I/O APIs under #[cfg(not(target_arch = "wasm32"))] and #[cfg(not(target_arch = "wasm64"))]. That should cover most cases.

A request: Would you be interested to write a section or two in the README about compiling to wasm? I have written a couple of paragraphs basically saying that we can do it, but not much else in terms of how to actually set up the toolchain.

[schungx]

Just out of pure curiosity, what practical uses will Rhai be on a wasm target, since I suppose you can always use JavaScript instead. That is, other than building a Rhai Playground site...

[alvinhochun]

I'm currently gating all the file I/O APIs under #[cfg(not(target_arch = "wasm32"))] and #[cfg(not(target_arch = "wasm64"))]. That should cover most cases.

Just a note, wasm64 isn't currently a thing, not even sure if anyone is working on it now.

A request: Would you be interested to write a section or two in the README about compiling to wasm? I have written a couple of paragraphs basically saying that we can do it, but not much else in terms of how to actually set up the toolchain.

I would say that pointing to the Rust Wasm book should be fine. My hacked-together project is loosely based on it, except that mine is only designed for local test-running and have deliberately skipped over the Node.js/npm dependency. Actually using Rhai is no different from other platforms.

Just out of pure curiosity, what practical uses will Rhai be on a wasm target, since I suppose you can always use JavaScript instead. That is, other than building a Rhai Playground site...

I can think of cross-platform games and game engines. I don't know if it is really plausible to use the web browser's JS engine for scripting (that is, not for interacting with the HTML DOM and other Web APIs). And to use JS on a native target would require pulling in a JS runtime like V8 or SpiderMonkey.

[schungx]

I can think of cross-platform games and game engines. I don't know if it is really plausible to use the web browser's JS engine for scripting (that is, not for interacting with the HTML DOM and other Web APIs). And to use JS on a native target would require pulling in a JS runtime like V8 or SpiderMonkey.

Yes, agreed. If you need native + web, and portable scripts, then you basically need to embed a scripting engine into the wasm.

If you'd pull from my latest PR https://github.com/jonathandturner/rhai/pull/163

This version includes the alternative Instant implementation plus gating out some of the unsupported stuff. It builds cleanly on my machine with wasm32-unknown-unknown. Would appreciate if you can test it out and see if timestamp works...

[alvinhochun]

Weird, looks like the previous build wasn't actually optimized for size due to misplaced configurations. The one uploaded now is the actual size-optimized build (no LTO).

I'll give your updated branch a try later.

[schungx]

It runs primes.rhai for slightly over 4 seconds while native optimized runs it in around 1.9 seconds.

So it is around 2.2x slower than native, which is really not bad for size build and running in a browser!

[alvinhochun]

I uploaded a new build that includes support for timestamp() from your branch.

[schungx]

Works great. This version runs primes.rhai in 3.3 seconds, which is less than 2x slower than native!

speed_test.rhai runs in 1.1 seconds, which is again less than 2x slower.

[alvinhochun]

Now that Rhai is confirmed to work in web browsers, perhaps the making of an "official" code playground could be considered?

[schungx]

Now that Rhai is confirmed to work in web browsers, perhaps the making of an "official" code playground could be considered?

Yes, I thought about that too, but then we'd need to spice up a web page... Not that it is difficult, but time consuming to do a good job.

And nowadays any respectable playground won't do without syntax highlighting and squiggly red lines under syntax errors... So I'm not sure we want to go that way yet...

If you want to give it a shot, I'd gladly accept!

[schungx]

Actually a more urgent task is probably to do a better manual/documentation/tutorial site... Right now the README is getting quite long, and a "The Rhai scripting language" site is really overdue...

I'm still undecided on whether to use Jekyll or Hugo or some other site generator to do it in...

Do you know about these kind of things?

[alvinhochun]

Not really, but I would try mdBook since a lot of Rust projects are using it for their documentations.

[schungx]

Wonderful. I'll look into it.

[schungx]

Thanks, @alvinhochun mdBook turns out to be exactly what I need to make The Rhai Book!

Closing this now since WASM is confirmed to be working.