Develop a plugin interface for the CLI

[alexcrichton]

Currently the wasmtime CLI comes pre-bundled with WASI functionality and optionally enabled proposals such as wasi-{http,nn,...}. Beyond this though that's it, there's no further abilty to load host funtionality into wasm modules. A question that's been asked in the past of us is if it's possible to have a plugin-style functionality for loading new host interfaces into the CLI for wasm modules to use. Historically this has been a question that hasn't really had an answer as it's not clear how best to implement this. Nowadays though with the component model this is at least more clear to me and I wanted to file an issue for this. This is a bit of a pie-in-the-sky issue where it's probably good to collect discussion on.

Another motivation I would have for functionality like this is that it makes it easier to produce a minimal Wasmtime executable. Producing a minimal executable can be helpful from the perspective of evaluating Wasmtime where CI artifacts can be reviewed to determine if Wasmtime is suitable for a particular use case or not. Currently the Wasmtime CLI "min" executable artifact contains all of WASI, but it doesn't make sense to remove it really because then you can't really do anything with the resulting executable other than look at it. With a plugin-style interface though it would be possible to use the "min" executable, provide custom host WASI implementations (or totally custom APIs), and get a working system out of it. Even if it's not the final working system that an embedding settles upon this could be an important milestone in terms of developing and proving something works.

Overall I don't at least personally have a super-well-fleshed out idea of where to take this. One rough idea is that WIT can serve as a good basis for the component model at least. For example one could imagine that given a WIT file a C header file is generated which represents the plugin API that Wasmtime expects to dynamically load. (or something like that) For core wasm it could be something similar-ish but with a direct translation from core wasm signatures to C signatures. This would still be significant chunk of work though to design further than this.

In any case I wanted to write down some thoughts on this, especially in relation to the "min build" use case. If others have thoughts or use cases I hope to collect them over time here as well.

[abrown]

I remember talking to @sunfishcode a long time ago about this — I'm in favor! It's tricky to do this well, though...

[sunfishcode]

I agree a plugin system would be very useful. A concern with dlopen-style plugins is that there's no isolation between the plugin and the host application. While there may be use cases ok with accepting the risks, I'd like to propose that Wasmtime first pursue an RPC-based plugin system first.

Running plugins in a separate process opens up options for sandboxing them using mature OS-based sandboxing technology, so it would be a great complement to Wasmtime's overall security story: the Wasm is sandboxed, and the host plugins are sandboxed. And, the ability to route RPC protocols across over networks would also open up new possible use cases. (We wouldn't need to implement either of these up front.)

A third benefit would be that it'd avoid the need to design a stable C ABI for plugins. Making a C ABI extensible is an art form, while protocols such as gRPC offer a variety of facilities for making APIs extensible.

A downside to RPC-based plugins would be performance. I imagine host modules that need the best performance possible will always want to be directly linked to the embedding API, which will likely be even better than a dlopen system could be. And over time, we could add shared-memory options to the RPC protocol to reduce protocol traffic when the plugin is local.

[abrown]

I would imagine most (if not all) host modules expect "the best performance possible." Seems like one would need to start with shared memory...

[bjorn3]

Even shared memory is rather costly when you are not doing asynchronous ipc like io-uring and virtio do.

[cfallin]

A slightly blue-sky idea: maybe one way to think of modularity here is to push as much into Wasm as possible (virtualization, but to the extreme). How well would it work to build most of WASI in Wasm modules and then provide raw Linux syscalls as Wasm imports to the lowest layer, which must be trusted?

[cfallin]

(As a side-effect, that would also make much of the "Wasmtime" build be platform-independent; the runtime itself would be the only native code...)

[bjorn3]

At that point I don't think you are winning anything in terms of safety and using linux syscalls will make it not platform-independent at all. Emulating linux syscalls on windows will be a pain and likely to introduce more security issues than it solves. You also can't directly export windows syscalls on windows, you have to expose the api's of the system libraries which often takes pointers.

[cfallin]

Well, we wouldn't provide Linux syscall imports on Windows; we would provide Windows API (edit: not syscall) imports, and the lowest-level modules would I guess be platform-dependent in that sense (but not ISA-dependent). The point isn't security, the point is modularity: given that we want to plug in various pieces and get to a core that is very small, and given that we don't want a native-ABI plugin interface, the thought is, why not use the stable boundary (namely, Wasm function calls) that we do have?

[bjorn3]

If you are compiling for a native target, the standard library and a lot of crates will handle all different platform api's for you. When compiling for wasm with those platform dependent api's, you will have to write all that yourself. Also even when using the platform api's from wasm, wasmtime would still have to contain a lot of bindings to adapt between wasm and the platform api's as wasm is unable to represent a lot of native C abi details like struct arguments and pointers into the host address space.

[cfallin]

Hmm, yeah, I suppose I was assuming that we'd lift any supporting library into Wasm as well. But perhaps this is less practical than @sunfishcode's multiprocess isolation approach.

[alexcrichton]

At the last CG summit I talked with some folks from CMU who are working on WALI which is basically what Chris is describing, and I would agree with the downsides @bjorn3 is describing as well.

One of at least my personal motivations for this would be fo reducing the binary size of the main wasmtime executable (or at least making it possible to do so), and one of the downsides of this approach is that we'd have a to have a function that translates all "wasm syscalls" to "native syscalls". For Linux that may be somewhat cheap as you only need to translate pointers but for Windows that'd probably be much more expansive since the set of APIs is quite large.

One mild worry about a plugin architecture being based on something like this is that it may run the risk of being limiting as well. If a plugin needs to reach into glibc for some bit of functionality such as pthread_create rather than the kernel then a syscall layer wouldn't easily expose the ability to do that. Plugins for example couldn't bring in their own native dependencies like OpenGL (pulling a thing out of the hat).

So I suppose after writing this down while I like the idea of further sandboxing that definitely comes with its own set of tradeoffs which may be interesting to work with. I'll definitely admit "add a plugin system" is so broad and undirected it's basically not actionable, so thinking some of this through makes me think that if we're too add a plugin system we should first decide on requirements for it. For example if a requirement is to be able to pull in native libraries off-the-shelf then this style of virtualization is a no-go. If that's not a requirement though then this is still a possibility. If a requirement is to have a very low binary size impact on Wasmtime ... (you get the idea).

As for IPC-vs-dlopen I think that also probably depends a lot on the requirements for what we'd want out of a plugin system. Having never developed a plugin system myself I don't actually know how we'd best go about even selecting the design constraints much less possible solutions.

[jeffparsons]

I'd like to propose that Wasmtime first pursue an RPC-based plugin system first.

Running plugins in a separate process opens up options for sandboxing them using mature OS-based sandboxing technology, so it would be a great complement to Wasmtime's overall security story: the Wasm is sandboxed, and the host plugins are sandboxed. And, the ability to route RPC protocols across over networks would also open up new possible use cases. (We wouldn't need to implement either of these up front.)

This approach would also be useful for users of the Wasmtime crate, and might help in exploring the design space for an eventual standard "remote component" extension to the model. (Here "remote" just means not part of the same root component instance / store, not necessarily in a different process/host.)

I'm currently trying to work out a cheap and cheerful degenerate version of this for my own needs. One question I ran into was around resources: it seems it would be a lot of work to make remote resources work, and I'm not sure if it's even a good idea. So I've ended up instead building an "external" version of my interfaces that's designed to be called across different stores, and wrapped up in HTTP for remote hosts. I'd love up hear from anyone with similar use cases.

As for the "C ABI" approach, if crABI takes off, might that help in building an ergonomic + extensible interface? It seems to me that it tackles a lot of the annoying problems that Wasmtime would otherwise need to invent its own solutions to.

[fitzgen]

Something to keep in mind, given the recent CLI breaking changes fiasco: we will want to very clearly communicate the expected stability of these plugins and eventually commit to pretty strong stability guarantees.

[fitzgen]

One rough idea is that WIT can serve as a good basis for the component model at least. For example one could imagine that given a WIT file a C header file is generated which represents the plugin API that Wasmtime expects to dynamically load. (or something like that) For core wasm it could be something similar-ish but with a direct translation from core wasm signatures to C signatures.

This would be super cool, but I wonder if the canonical ABI (translated to native? handwaving) would be stable enough yet?

Also, how would we handle options like string encodings?

---

In the meant time, to have a stable interface, we could essentially support just the array calling convention and that's it. I feel pretty comfortable with supporting that long term, modulo figuring out the details of symbol names and all that such that we could start supporting a canonical ABI-based thing transparently in the future when newer symbols are named.

[Mossaka]

I remember we had similar discussion and a desire to add a plugin system to wasmtime during the BytecodeAlliance Summit early this year. I agree such a plugin system would be extremely useful from the perspective of adding wasi-cloud-core host APIs.

[squillace]

Two things.

First, I'm going to tag @shschaefer from the Windows team as we were discussing Windows syscalls on wasi built with component model just yesterday. Re: https://github.com/bytecodealliance/wasmtime/issues/7348#issuecomment-1777695608

Second, this is not a technical consideration but a business one, and presented for everyone's information. There are tons of cases in which Microsoft (and others) will want to circumscribe what can be run and ensure that by making those static, decided at compile time. HOWEVER, I can already say that we will in the future require the ability to have the runtime load the component declarations, perhaps in a config file, perhaps with a plugin, or another mechanism. (I'm not advancing any implementation, only the feature requirement.) This would eventually be such a significant blocker that we'd consider jumping runtimes and/or implementing. This isn't required "now" in any sense, but I already know that two years down the line might be too long to wait -- just to give people an idea. And yes, happy to help find resources.

[fitzgen]

@squillace

HOWEVER, I can already say that we will in the future require the ability to have the runtime load the component declarations

Can you clarify: you require this ability in the Wasmtime CLI, not the embedding API?

[squillace]

Cheers @fitzgen we'd eventually require it in all use cases. I should mention that because we see a lot of usage scenarios that widely vary, the idea that a CLI doesn't have the vast majority of the API possibilities is not one we hold. For example, internally we ship APIs for azure services but the CLI AND portal are all required to match the capabilities or the service cannot be supported (what we call GA). So the expectation is that at some point all "surfaces" to the functionality are equal with the sole exception being something that cannot be done (of which there are always some, of course). (To pre-answer a common question, yes usage varies widely, but it turns out that some of the least usage -- the portal -- has the highest satisfaction and adoption impact.)

I wouldn't expect anyone else to immediately agree with the principle! I just wanted to use it to help explain where we come from. In addition, there are always special cases and of course time taken to synchronize and resource acquisition to do so and so on. Happy to flesh this out more if that helps. Does it?