Add/test engine WASM friendliness

[AndrewSisley]

Would be handy (and fun) if the engine itself was able to run in WASM, I'm not sure there would be too much effort to permit it to do so.

The Go JSON stuff might be the only notable issue, or you could just write a Rust host (https://github.com/lens-vm/lens/issues/27) :)

[AndrewSisley]

The Go JSON stuff might be the only notable issue

As of 2023-11-01 there is no way to compile Go to just WASM, using either the standard Go compiler or TinyGo. You can compile to WASI (which results in a lot of bloat and hassle due to mandatory imports), or you can compile to a fairly unpleasant kind of WASM designed to be used via compiler generated JS bindings (also lots of bloat and mandatory imports). Both of these are undesirable.

Writing a Rust engine that can also compile to WASM still seems the most sensible thing to do. Although it come with a bit of complication on the Defra side (we want to be able to compile Defra to WASM, and still use Lens), as instead of native Go bindings it will have take the Lens engine as an (wasm) import.

Because of this it may even prove sensible to have native-Defra use the wasm Lens engine instead of a Go native one. That way Lens in both native and wasm Defra can work in largely the same way, and share the same code (the only difference being how the engine is initialised and passed into Defra's internals).

Lens could then actually be authored wasm-first, and offer specific language bindings that sit on top of the wasm version (like how Defra's python sdk sits on top of the http/openAPI client).

Regardless of implementation language, I believe the engine architecture will need to radically change.

As modules must be imported into the engine, and there is no way to import an array of functions, engine instances will need to wrap each module (or its wrapping engine instance) creating a stack of engine-module instances:

--engine----engine-----engine--
     \\         \\          \\
   --module1-  -module2-  -module3--

This will likely result in an increased memory overhead per transform. There may also be a per-item performance cost in transferring between module and engine instances, although hopefully it is possible to avoid this.

It may be possible to avoid this layered approach by metaprogramming and compiling (from implementation language to .wasm) in the consuming system once the lens modules have been defined, this would allow for a much leaner critical path (when processing items), but with a huge host of other drawbacks.

[AndrewSisley]

I've spent a fair amount of time staring at this issue the last few days, and believe that a lens 'engine' cannot be made wasm friendly, as pretty much all it does is handle the communication across wasm module boundaries - something that needs to be done in the host runtime (it can also happen between modules, but the host code still needs to exist anyway, and the host might as well handle both).

When thinking of Lens within a wasm environment, what is really needed is native engines in the host lang (e.g. JavaScript in a browser). These may need slightly different interfaces than the current, as they may need to protrude deeply into the wasm space (e.g. DefraDB).

As a result, this ticket should probably be closed long-term, but I'll leave it open for now in case anyone (including future me) thinks otherwise.