WASM support

[andychu]

Why? On first glance this is probably expanding the scope of the project too much. But:

• Like headless mode, it allows pushing some functionality out of the core of Oil, and into plugins. So in a sense we can reduce scope (although it's likely to be significant work).
• We need fast in-process string transformations for string safety #938
  • shell is not a good language for writing this, and we don't have Tea yet.
• Other motivation: for dev tools bootstrapping #802
  • to solve the portability problem
  • e.g. custom versions of libcmark and re2c
  • re2c probably needs too much file I/O though?

Design: Two Types of Plugins

• inline (no fork).
  • read(0, &buf, N) and write(1, buf, N).
• pure proc-like.
  • but NO FILE SYSTEM. I want this to figure into the dependency engine / Ninja
• much later, or never: full WASI

Which Engine?

• wasm3 uses unportable tail calls, although maybe there's an #ifdef you can use for it
  • try compiling it for some odd architecture??
  • note: see lobste.rs exchange about C API

Crazy Idea

• maybe our sqlite integration can live an wasm plugin. This would slow things down, but maybe that doesn't matter...

Related to #848

[andychu]

Some more motivation here: https://lobste.rs/s/qjeefx/wasm3_compiles_itself_using_llvm_clang#c_dpuuhg

Actually I have an idea to embed WASM in https://www.oilshell.org/ . One use case is to solve some bootstrapping problems with dev tools. For example, if you have a tool written in a native language like a parser generator, then it’s somewhat of a pain for people to either build those, or for the maintainer to distribute binaries for them (especially if they change often).

So it seems natural to write a shell script and call out to an “embedded” arch-independent binary in those cases. (Though this probably won’t happen for a long time.)

Problems with WASM

• provenance of source code. I don't want unreadable binary blobs.
  • however, solving this provenance problem also relates to Oil's other goals, so it could be worth doing
• speed -- if we're sticking to the principle that we should distribute portable C++. That means using a WASM interpreter, and the fastest WASM interpreter uses unportable C++ (wasm3 tail calls).
• probably bloated toolchains and dependencies
  • lack of shared libraries will lead to bloat, e.g. for common utf-8 string processing routines

[vshymanskyy]

@andychu you're referring to some "odd" architectures. Please let me know what exactly you have in mind, as Wasm3 was verified to work on 10+ architectures (even Big-Endian) without issues.

[andychu]

Well it's more of a philosophy to have the Oil core 100% standards compliant. That way you get ALL architectures and not just 10+.

It might be worth breaking that rule for some killer feature -- maybe fast WASM is it -- but I'm not sure yet.

Are there any new docs about how to embed wasm3? That would be helpful

[andychu]

I guess the obvious solution is to provide a conditional compilation option, e.g. for which WASM engine to use, just like you can compile different malloc() in or whatever.

bash can be compiled with or without GNU readline, and so can Oil.

We could do that for Oil and wasm, but it would be nice to avoid fragmentation. That is if it supports wasm, then it would be nice if people can rely on it everywhere

This is a little hypothetical, since I think the wasm work is a little far in the future. Our plate is overflowing with things to do now!!

[vshymanskyy]

Embedding is easy, just include Wasm3 sources in your build system, and use any of the examples (I suggest to start with Arduino examples, i.e. run fibonacci function).

On Tue, Jun 15, 2021, 3:38 AM andychu @.***> wrote:

I guess the obvious solution is to provide a conditional compilation option, e.g. for which WASM engine to use, just like you can compile different malloc() in or whatever.

bash can be compiled with or without GNU readline, and so can Oil.

We could do that for Oil and wasm, but it would be nice to avoid fragmentation. That is if it supports wasm, then it would be nice if people can rely on it everywhere

This is a little hypothetical, since I think the wasm work is a little far in the future. Our plate is overflowing with things to do now!!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/oilshell/oil/issues/941#issuecomment-861083183, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALP3FEZXJJKS2I5REEU2ADTS2OIJANCNFSM444DI4ZQ .

[andychu]

Hmm this is now out of scope; we're more focused on containers #848