search

helix-editor / helix

A post-modern modal text editor.
https://helix-editor.com
Mozilla Public License 2.0
33.97k stars 2.51k forks source link

WebAssembly plugins system #122

Closed CBenoit closed 7 months ago

CBenoit commented 3 years ago

Basically load .wasm files.

Capabilities:

At first we could use a basic toml config file or CLI to feed .wasm files to the editor.

A way to configure permissions on a plugin basis could be investigated to use the sandboxing capabilities coming with WASM. Example with wasmtime:

$ wasmtime --dir=. --dir=/tmp demo.wasm [args…]

(reference)

I think the biggest challenge is to get well-defined interfaces down but let's not fear to break it during early stages.

Later we could investigate embedding a wasm-based scripting language such as Grain or AssemblyScript.

Here are some references:

I'm willing to experiment soon

archseer commented 3 years ago

I think the most basic demo would be implementing one of the existing commands in such a way, e.g. https://github.com/helix-editor/helix/blob/407b37c3279bfd0ae2bf756bc022d47d5db446d9/helix-term/src/commands.rs#L123-L137 It only depends on the Context, and it only uses functions from core (move_horizontally) without interacting with the UI. I think most of the primitives in core should be exposed (selection/range/transaction/etc) and we can have a subset of view exposed (retrieving a selection from a doc, applying transactions)

archseer commented 3 years ago

Mentioned on Matrix:

Re: wasm plugins, rich clipboard support would be a great demo and to that end, https://github.com/neovim/neovim/issues/14706 "[RFC] Neoclip: multi-platform clipboard provider w/o extra dependencies" may be worth watching.

buster-blue commented 3 years ago

What benefit does this provide over binary plugins? Will plugins be able to access eachother's state (functions, variables etc.) in the same way that plugins programmed in a single language (e.g. emacs) are able to, or something else?

My bad if the answer is obvious btw. I'm not very familiar with WebAssembly or plugin architectures.

kirawi commented 3 years ago

What benefit does this provide over binary plugins? Will plugins be able to access eachother's state (functions, variables etc.) in the same way that plugins programmed in a single language (e.g. emacs) are able to, or something else?

My bad if the answer is obvious btw. I'm not very familiar with WebAssembly or plugin architectures.

I think this post explains it well. There's a lot of uncharted ground in regards to plugin systems, especially with sharing state, but you can see past discussion on this here.

CBenoit commented 3 years ago

@kirawi Thank you for the link to Valoren, I missed that one

matu3ba commented 3 years ago

If you want to embed the wasm compiler with the binary or as library etc, I would advise against wasm due to slower speed and missing performance optimisations on many targets.

"Of course, wasm3 runs fine on aarch64, but at about 3% of native speed."

This benchmark shows very astonishingly the missing optimisations that luajit can do here using this lua libary.

Also it would be advisable to have execution time benchmarks of webassembly against all usable alternatives before making a decision (ie by listing some speed and platform compatibility requirements). Would be not very useful to end up with something significantly slower than neovim.

kirawi commented 3 years ago

If you want to embed the wasm compiler with the binary or as library etc, I would advise against wasm due to slower speed and missing performance optimisations on many targets.

"Of course, wasm3 runs fine on aarch64, but at about 3% of native speed."

This benchmark shows very astonishingly the missing optimisations that luajit can do here using this lua libary.

Also it would be advisable to have execution time benchmarks of webassembly against all usable alternatives before making a decision (ie by listing some speed and platform compatibility requirements).

The intention is to use wasmer or wasmtime which are both written in Rust, and are quite fast in the benchmarks. However, real-world use is not going to be accurately represented by benchmarks. If it wasn't fast, it wouldn't be used by Google or Figma to power some of their most resource-intensive client-side services, nor would it be able to power game engines on the web. (Here is a more complete list.)

Lua lacks multithreading, SIMD, and has a smaller ecosystem. It's a lot more convenient to enable users to opt for their favourite language rather than have them learn a separate language and its ecosystem. I can't speak on behalf of everyone, but I believe that most Rustaceans firmly agree with me, as many libraries have wasm as a compile target. I would say we collectively have a lot more experience with wasm as a whole.

CBenoit commented 3 years ago

We'll probably go with wasmtime and I can see in one of the benchmark you linked that performances are far from bad. By the way wasmtime has been supporting WASI for a while prior to the article publication date, but they don't mention it, so I assume they used old data from a previous benchmark,

Older versions of the following runtimes had been tested in the previous rounds: […]

Given that Cranelift code generator is still very young and already that good, I'm not too afraid. Furthermore performance tracking is a thing at the bytecodealliance.

Besides all the other good points made by @kirawi on performance, I really like the ecosystem argument which is the reason why we want to use WASM in the first place. Given that performance is probably good enough (or more than enough), the main focus should be on whether user/developer experience is good.

syrusakbary commented 3 years ago

Thanks for considering Wasmer! We are working on a set of benchmarks at the moment, we had two bugs that prevented Wasmer to shine in the one you showcased @CBenoit, but those are already solved in master.

Right now Wasmer is about 20~30% faster than the other runtimes when using the LLVM compiler!

In general, we recommend Cranelift if the compilation times need to be fast (that is mainly for development), and LLVM for production (similar to how Rust uses LLVM always to compile in release mode, and Cranelift as an experimental compiler for faster compilation times in debug mode) :)

kirawi commented 3 years ago

Thanks for considering Wasmer! We are working on a set of benchmarks at the moment, we had two bugs that prevented Wasmer to shine in the one you showcased @CBenoit, but those are already solved in master.

Right now Wasmer is about 20~30% faster than the other runtimes when using the LLVM compiler!

In general, we recommend Cranelift if the compilation times need to be fast (that is mainly for development), and LLVM for production (similar to how Rust uses LLVM always to compile in release mode, and Cranelift as an experimental compiler for faster compilation times in debug mode) :)

This is a question I've had for a while, but what is the main difference between wasmtime and wasmer? I can't recall exactly, but I believe they're targeting different use cases, correct?

CBenoit commented 3 years ago

Thank you @syrusakbary for your input!

This is a question I've had for a while, but what is the main difference between wasmtime and wasmer? I can't recall exactly, but I believe they're targeting different use cases, correct?

I'm interested in hearing more as well!

I was more in favor of wasmtime for the following reasons:

On the other hand, wasmer:

I'll also ask on Veloren's Discord why they decided to use wasmer to form a better opinion.

I also found this article published a few months ago:

  • Wasmer has the best overall support compatibility with every programming language at super-speed
  • Wasmtime is lightning-fast and compact, with good configurability but fewer languages supported
  • Lucet is a specialized solution for running untrusted WebAssembly programs inside a larger application
  • WAMR runs with a small footprint
matu3ba commented 3 years ago

@CBenoit @kirawi There are 2 mutual exclusive possible use cases for WASM for plugins, where one really wants to have a plugin manager: 1. shipped compiler or 2. hook into the repo build system and the used compiler to build the WASM libraries/binaries.

  1. interpreted which one usually wants to ship a dedicated compiler (which is language-specific for performance => This is effectively a soft language-lockin as one wants to ship it with the editor and not bundle all compilers/programs that can emit WASM.

  2. compiled (fastest) which can be completely independent, but requires support of 1. the build system and 2. distribution package managers etc to have the compilers installed. If the language does not support easy build system integration and there is no convention where to put stuff upstream, it does scale poorly (because the package manager of helix can not build stuff).

Solution 1. gives more of a "it always works" experience with the language that the compiler supports

Solution 2. gives more of a "its super fast", but needs some setup and might break. Unless you can use nix for 2. ie with flakes 2 is just painful due to 2.1 many distros not shipping all compilers or only specific versions and then you are stuck on one (ie not yet stable languages) and 2.2 you want to simplify build system stuff for plugin developers.

nix with flakes is the de-facto best solution for 2.1 (shipping compilers+libc(s)? and make sure they exist, are configured/build identically and up-to date enough for the plugin ecosystem). For 2.2 nix flakes should work, but a convention how build commands/shell scripts are supposed to be called could also work (I am not sure how stable flakes are for that use case for all the build systems that exist).

Ask yourself which solution is better for the goals of the project helix or how you want to deal with the mess of 1. shipping compilers and 2. supporting the build systems of plugins.

kirawi commented 3 years ago

We won't be shipping a compiler, though? WASM is a compile target for Rust itself as wasm32-unknown-unknown, and other compilation methods use their own external tool (Wasmer and Wasmtime also provide their own) to compile it down to .wasm, which would be run by the WASM runtime. Languages like Python or Lua currently work by compiling their interpreters down to .wasm, but in the future they could be directly compiled down to .wasm once the WASM GC is standardized and implemented.

Did I understand you correctly?

matu3ba commented 3 years ago

@kirawi How does this change the problem, when the plugin is written in a language that the distribution doesnt ship a compiler (or super old/incompatible or slow libc etc) ? The plugin then just doesnt work? Or works differently and the plugin author should deal with the mess of different compiler versions?

kirawi commented 3 years ago

No, it's just .wasm. The user doesn't need to download anything else other than that standalone .wasm output which is what the plugin actually is. Think of it as creating an executable, you don't need all the code and compiler stack to run the executable as an end-user, because the OS already has everything it needs to run the executable inside the executable itself. The executable in this case is .wasm, and the OS is the Wasm runtime. Wasm is completely language agnostic and is just bytecode.

matu3ba commented 3 years ago

@kirawi Thanks alot for the clarification.

How is 1.integrity (nobody tampered with the build process/environment), 2. debuggability/tracebility (what stuff got wrong) and related 3. uniformity (potential contributors can reproduce the .wasm to manually reproduce/fix stuff)?

Point 1. is a stopper for usage in any more security-critical developer environments, point 2. is okayish not to have (debugging luajit also doesnt work very good), point 3. can be super annoying to find contributors.

kirawi commented 3 years ago
  1. Couldn't you argue that for any software? If you're worried about it, you can compile the plugin yourself.
  2. Wasmtime and Wasmer.
  3. The potential contributors would need to know the code, but that's the same for any language.

The same challenges exist for any language, but Wasm is a compile target, not a language, allowing for anyone to write any plugin in any language they want, instead of restricting them to one.

matu3ba commented 3 years ago
  1. I strongly disagree on 1 reproducable builds, since this gives to much trust to single developers to do potential harm while making it hard to detect it. You can not reproduce a binary/wasm file without an exact identical build environment. Also you may have to deal with third parties shipping binaries etc.
  2. thanks alot.
  3. Conventions and build systems influence how code is structured, but that is overall correct.
kirawi commented 3 years ago

Indeed, but that is something that is just going to happen no matter what. Every program that people use is going to be written in separate languages, and if you want to do that of risk analysis, you would want to compile it yourself. But the other benefit is that since you can write a plugin in your preferred language, you can for the most part write all your plugins in-house. Lua is arguably worse because it lacks sandboxing, while with Helix and Wasm we would have full control over what a plugin can or cannot do.

It's a tradeoff for sure, but I think Wasm opens up a lot of possibilities to circumvent the cons of using Wasm.

matu3ba commented 3 years ago

@kirawi How much performance loss does sandboxing mean?

L3 cache can likely never fully be mitigated without notable performance loss, since it is shared between cores (and mitigation would require accurate time synchronisation between memory controllers of cores).

Here is the list of possible cache attacks which looks scary to me. And on top of that: "most existing hardware is broken in terms of security! And the underlying cause is the ISA, which is our hardware-software contract." risc5 timing channel analysis paper should be linked in the article => flushing might not flush cache immediately.

kirawi commented 3 years ago

Wasmer and Wasmtime are both sandboxed, so you can reference the earlier benchmarks for them. Admittedly I'm not sure on the cache aspect, and I'm aware that Wasm sandboxing is far from perfect, but I think that leads back to the point that you would need to evaluate the plugin if you're untrusting of the plugin. I also found this article, maybe it'll shed some light.

Just being on the browser right now, you're probably running quite a lot of JavaScript and Wasm already. I think this discussion, though important in general, is bike shedding in this context. There isn't a perfect solution here, and there probably never will be, but Wasm seems like the best option for Helix.

pickfire commented 3 years ago

Can we have two in one option? On one hand we let developers have it compiled so it could be faster. On the other hand we can have it interpreted to allow testing stuff faster. We could use the interpreted version as one of the compiled plugin. Not sure how feasible is this.

CBenoit commented 3 years ago

That's definitely some workflow we could think of at some point. Makes me think of the gcc-emacs that is compiling elisp instead of interpreting it IIRC.

I prefer investigating the embedded runtime approach fully before we consider that though.

vshymanskyy commented 3 years ago

Just for clarification. Wasm3 is 4-5 times faster than, say, Python 3.9, yet weights only ~150KB. It doesn't require any JITting, etc. so runs exactly the same on iOS, Android, Windows and other platforms.

On Apple M1 for instance, it showed ~5-7% of native speed, which is somewhat better than 3% that was claimed here. But this wasn't heavily tested/optimized anyway.

archseer commented 3 years ago

Today in the news: https://wasmer.io/posts/wasmer-2.0

Seems to have some support for reference types. I'd still like to see a comparison with wasmtime beyond just performance.

(Edit: Nice, I see the announcement is written by @syrusakbary himself)

tschneidereit commented 3 years ago

Hey all👋🏻

I work on Wasmtime (I manage the Fastly team doing much of the development work), and represent the Bytecode Alliance, so I don't think it'd be appropriate to chime in with a direct comparison between the two projects. Instead, I want to offer conversations to help answer questions you might have—feel free to shoot me emails ([my gh handle]@fastly.com) or a DM on the Bytecode Alliance Zulip.

A few pieces of information about Wasmtime that might be relevant to your evaluation:

malte-v commented 3 years ago

Another idea: Do all configuration and plugins prior to compiling similar to how the suckless people do it. The end-user would clone a repo/crate that depends on the helix crate and acts as a configuration template. The editor would be configured by filling in some structs and passing a Config object to an initialization procedure from the helix crate. Similarly, plugins would just be crates each exporting some kind of plugin definition, which would alse be passed to the initialization procedure of the helix crate.

Pros

Cons

sudormrfbin commented 3 years ago

Another idea: Do all configuration and plugins prior to compiling similar to how the suckless people do it.

Some concerns about this approach:

malte-v commented 3 years ago

Scripting languages are better suited for tasks like this, and locking it into a language has a steep learning curve might not be the best decision.

Most scripting languages rely on a garbage collector. Does Wasm support this? How should we handle objects created on the GC side and passed to Rust? The plugin author would have to tell the GC to forget about objects, which is neither user-friendly nor particularly safe.

If helix went a step further and allowed writing plugins in any language and still get the same perfomance (considering the wasm overhead of course), that'd be a game changer.

If helix allowed writing plugins in any language, then the plugin API could at most use features that are available in any language and are also available in the Wasm ABI. Does Wasm support closures? Dynamic dispatch? (Genuine questions, I don't really know much about Wasm.) I think supporting every language around will make the plugin API very limiting.

My main concern is the amount of glue code required to make Wasm plugins (let alone non-Rust ones) work. I would say that Rust is a pretty mainstream and widely used language nowadays, so I wouldn't worry about the user-friendliness of Rust too much. On the other hand, setting up a Wasm build environment for a language that doesn't have great support for it often nontrivial and might put off plugin authors.

CBenoit commented 3 years ago

Regarding garbage collection, currently some scripting languages implement it manually like AssemblyScript, and yes there is a proposal for it. Obviously, we should aim for user friendliness a minimum here. However I don't think what you cited will be issues, plugins authors will not have to tell the GC to forget about objects.

If helix allowed writing plugins in any language, then the plugin API could at most use features that are available in any language and are also available in the Wasm ABI. Does Wasm support closures? Dynamic dispatch? (Genuine questions, I don't really know much about Wasm.) I think supporting every language around will make the plugin API very limiting.

Obviously API will be limited to what WASI allows us to do. We know that, it's okay. It's a tradeoff.

My main concern is the amount of glue code required to make Wasm plugins (let alone non-Rust ones) work. I would say that Rust is a pretty mainstream and widely used language nowadays, so I wouldn't worry about the user-friendliness of Rust too much. On the other hand, setting up a Wasm build environment for a language that doesn't have great support for it often nontrivial and might put off plugin authors.

Glue code can be part of a library. We'll probably provide Rust one and a few others. Then yeah, you're right it'll probably be harder for a language with bad wasm support, but… should we care about that? I don't think so. Languages with good wasm support are numerous and in a growing number anyway.

kirawi commented 3 years ago

https://github.com/appcypher/awesome-wasm-langs Many GC languages that can run on Wasm currently do the same as what @CBenoit said.

syrusakbary commented 3 years ago

I work on Wasmtime (I manage the Fastly team doing much of the development work), and represent the Bytecode Alliance, so I don't think it'd be appropriate to chime in with a direct comparison between the two projects. Instead, I want to offer conversations to help answer questions you might have—feel free to shoot me emails ([my gh handle]@fastly.com) or a DM on the Bytecode Alliance Zulip.

@tschneidereit that's an awesome way of handling this. Chapeau for it!

If anyone wants to ask specific questions about Wasmer and the tradeoffs we developed upon please feel free to do the same at syrus@wasmer.io or chime in on Wasmer community slack

I believe the Helix project will be in good hands if they choose either project. Both are handled by very passionate people in the WebAssembly industry and we care deeply to continue moving it forward 🤗

CBenoit commented 3 years ago

Thank you very much! @syrusakbary @tschneidereit I think I'll drop by to see you both and ask more questions soon :slightly_smiling_face:

kirawi commented 3 years ago

MVP #455

TheLostLambda commented 3 years ago

Heya! Just figured I'd also drop in to say hello here! I'm the primary author of Zellij's plugin system and recently stumbled across Helix! I'm more than happy to chat with people about how our system is implemented in Zellij and how it could be used to inspire something similar in Helix.

I suppose the only thing we're doing that is somewhat "unconventional" is using WASI's stdin and stdout pipes to facilitate JSON communication. I helped expose this Pipe object in Wasmer, but it looks like wasmtime-wasi has something similar?

This JSON approach was really nice because, paired with serde, we could effortlessly pass complex data objects over the plugin-host boundary without ever needing to faff about with direct memory manipulation and pointers.

All said, the WASI pipes and JSON approach allows plugins to send and receive rich structs and enums that any JSON-aware programming language can understand. With a thin shim library, plugins can look like totally normal code without much to indicate there is WASM magic going on. The minuscule size of the shim library also makes it trivial to introduce first-class support to a number of other programming languages (including scripting languages like lua).

Regardless of how things evolve here, I'm very excited to see this feature implemented in Helix and am more than happy to offer whatever help I can along the way! Feel free to ping me with any questions and good luck!

kirawi commented 3 years ago

Heya! Just figured I'd also drop in to say hello here! I'm the primary author of Zellij's plugin system and recently stumbled across Helix! I'm more than happy to chat with people about how our system is implemented in Zellij and how it could be used to inspire something similar in Helix.

I suppose the only thing we're doing that is somewhat "unconventional" is using WASI's stdin and stdout pipes to facilitate JSON communication. I helped expose this Pipe object in Wasmer, but it looks like wasmtime-wasi has something similar?

This JSON approach was really nice because, paired with serde, we could effortlessly pass complex data objects over the plugin-host boundary without ever needing to faff about with direct memory manipulation and pointers.

All said, the WASI pipes and JSON approach allows plugins to send and receive rich structs and enums that any JSON-aware programming language can understand. With a thin shim library, plugins can look like totally normal code without much to indicate there is WASM magic going on. The minuscule size of the shim library also makes it trivial to introduce first-class support to a number of other programming languages (including scripting languages like lua).

Regardless of how things evolve here, I'm very excited to see this feature implemented in Helix and am more than happy to offer whatever help I can along the way! Feel free to ping me with any questions and good luck!

Was there a reason why JSON was preferred? I would have thought that something like protobuf would be a better choice.

pickfire commented 3 years ago

Was there a reason why JSON was preferred? I would have thought that something like protobuf would be a better choice.

Protobuf not sure but I think capn proto or flatbuffers might be better.

If we only support rust then https://github.com/djkoloski/rkyv might be a good idea since support for other languages isn't there yet.

kirawi commented 3 years ago

Protobuf has pretty good support in other languages: https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats Maybe flatbuf would be better since it supports languages most people will know, and it seems to be better than protobuf.

syrusakbary commented 3 years ago

rkyv has a great support for serialization/deserialization Rust, we use it heavily at Wasmer. We tried flexbuffers (a derivative from flatbuffers) but was not as fast in our testings.

Adding @djkoloski (from rkyv) to the thread as he might be able to help in some way!

kirawi commented 3 years ago

I'm not sure, I think we could switch to rkyv in the future when/if it supports other languages, but I think limiting ourselves to Rust sort of defeats the main value prospect for a Wasm plugin system. We're not guaranteeing backwards compatibility yet so it shouldn't be an issue to switch in the future.

archseer commented 3 years ago

I think rkyv is great, I've used it on work projects before.

That said, I think it would be good to avoid ser/de-ing data altogether, @CBenoit and me were leaning towards witx-bindgen which was mentioned earlier in the thread.

kirawi commented 3 years ago

It does seem like an increasingly better option for this kind of use case.

syrusakbary commented 3 years ago

Yeah, I think witx-bindgen (on their latest version of the syntax) is great. The downside is that you still tie yourselves on a C-ABI-like memory layout, but I think that's probably an acceptable tradeoff.

I see that in the prototype you are not using Wasmer, but we will be super happy to help and assist with any integration if you end deciding that Wasmer is the way to go :)

djkoloski commented 3 years ago

Sorry for the delay, I've been catching up on the content of this thread and doing my best to really understand the issues at hand. Just to be clear, I am the author of rkyv so my perspective will be biased.

I agree with @archseer that ideally no ser/de work would be the optimal outcome. All of the data being passed around is kept in-memory and if you really want a language-agnostic API then you'll need to create bindings anyway. It may be prudent to limit plugins to rust if they need access to deep engine internals like raw data formats.

I always look for opportunities to help people use rkyv, so if you do try it out and want any help or advice, feel free to message me directly or drop by the rkyv discord.

TheLostLambda commented 3 years ago

Was there a reason why JSON was preferred? I would have thought that something like protobuf would be a better choice.

Purely for compatibility and simplicity! Any language with WASI support would almost certainly have the ability to read JSON. Since the host program is Rust though, switching to any other serde-supported format is trivial (we pondered MessagePack for a while).

For us, the flexibility and accessibility of the plugin system was the main point of using WASM, everything else came second :) Honestly though, if it's a format that most-all languages support, it doesn't really matter what you choose!

If we only support rust then https://github.com/djkoloski/rkyv might be a good idea since support for other languages isn't there yet.

Thanks for pointing me to rkyv! It looks super cool! I agree with @kirawi though that is kinda defeats the point of using WASM if we lock ourselves into just one language.

That said, I think it would be good to avoid ser/de-ing data altogether, @CBenoit and me were leaning towards witx-bindgen which was mentioned earlier in the thread.

witx-bindgen does look really cool, though it does limit the possible plugin languages a little. Also, I may be misunderstanding how things work, but would it only allow plugins in either Rust or C at the moment? I know it can create JS bindings, but as I understand that JS still needs a runtime? Potentially something as heavy as Deno? Again, I could be misunderstanding this!

I think I saw scripting languages mentioned somewhere farther up the thread and I do think it would be nice to support a couple if possible. I think with the WASI solution, we'd just need some interpreter that runs in WASI, but I think we'd have to wait for witx-bindgen to implement more languages otherwise?

kirawi commented 3 years ago

I believe that languages could use the C bindings, as C is almost universally supported through FFI. Hopefully cleaner bindings could be built on top of that.

CBenoit commented 3 years ago

I see that in the prototype you are not using Wasmer, but we will be super happy to help and assist with any integration if you end deciding that Wasmer is the way to go :)

Thank you! To be honest the decisive factor was notably witx-bindgen, because I want to play with it early but as an early prototype only wasmtime is supported as code generator host target. I hope to see Wasmer as a potential target too in the long term :slightly_smiling_face:

That said, I think it would be good to avoid ser/de-ing data altogether, @CBenoit and me were leaning towards witx-bindgen which was mentioned earlier in the thread.

witx-bindgen does look really cool, though it does limit the possible plugin languages a little. Also, I may be misunderstanding how things work, but would it only allow plugins in either Rust or C at the moment? I know it can create JS bindings, but as I understand that JS still needs a runtime? Potentially something as heavy as Deno? Again, I could be misunderstanding this!

Already posted in the MVP PR, but just posting here as well for readability. Alex Crichton in bytecodealliance’s zulip said

On the wasm-content side, witx-bindgen and the general scheme around interface types and the canonical ABI is all being designed explicitly to support multiple languages on all sides, meaning that a wasm module using interface types can be written in any language and the runtime consuming the wasm module can also be any supporting runtime (e.g. wasmtime, a browser, node, ...). Currently I've implemented host support in the form of Rust wasmtime bindings and JS web/node/deno bindings. For the compiled-to-wasm side I've only implemented Rust so far, but I hope to implement at least C in the near future as at least a proof-of-concept. The intention is that writing a language binding generator isn't the hardest thing in the world (but also not a trivial task), so other languages can be done as they come up as well.

In short, witx-bindgen generates host glue code for JavaScript (Deno, Node, …). This is for executing WebAssembly inside runtimes such as Deno, not for writing WASM modules. I hope it clarifies :slightly_smiling_face:

I think I saw scripting languages mentioned somewhere farther up the thread and I do think it would be nice to support a couple if possible. I think with the WASI solution, we'd just need some interpreter that runs in WASI, but I think we'd have to wait for witx-bindgen to implement more languages otherwise?

As you pointed, at some point providing a few modules for interpreters running in WASI such as python and lua is a good idea! At least, it’ll be possible to provide such a plugin (basically wrap the interpreter and expose Helix functions to it AFAIK). We can also expect other kinds of scripting languages such as AssemblyScript (which is basically like TypeScript for WASM), or Grain that are targeting wasm directly. Generating bindings with witx-bindgen will depend on witx-bindgen, but even if the language one wants to use isn’t supported, user can declare functions to use directly (extern and such). The *.witx files are documenting the signatures in a language-agnostic way.

ym-han commented 3 years ago

Is it going to be at all possible to somehow use or adapt extensions from other editors (either from neovim or emacs or vscode)? I haven't really seen any mention of this point before, but I think there's something to be said about seeing if there's any way to capitalize on all the incredible work people have already put into plugins for other editors and ecosystems, if possible.

(Having said that, it looks like you're already pretty deep into designing the plugin system, so it might be too late to talk about this.)

kirawi commented 3 years ago

Haha, it's far from being too late. We're still in the pre-RFC stage to discuss requirements and implementation details. Feel free to chime in there :)

Right now might be the perfect time, actually. Me and @CBenoit are very busy at the moment, so progress is unfortunately moving at a snail's pace. Hopefully not for long, (though I'm not promising anything).

Serif-7 commented 3 years ago

I'm an inexperienced dev, so I'm probably missing a lot on this issue, but it seems to me like Lua is the obvious and probably best choice for helix plugins, for the following reasons:

  1. The most obvious is 'interop' between the Neovim dev community and the Helix dev community. There is a very large overlap between these tools, and it would make the future Helix ecosystem better if there was an easy pipeline between writing plugins for Nvim and writing them for Helix.
  2. Lua is simple, and designed to be embedded. @kirawi wrote a few months back that lua lacked SIMD and multi-threading, but do we really need either of those features for a text editor plugin? I don't think so. Lua is small and fast, and LuaJIT is even faster. Unless you just really want to play with WASM, I don't see why we shouldn't use Lua.

This might be totally pointless if you're all already building the wasm system though!