wasmtime serve: allow component reuse, serving > 1 request per instance

[ydnar]

Thanks for filing a feature request! Please fill out the TODOs below.

Feature

I’d like an option (or default) for wasmtime serve to reuse a component for multiple requests. e.g. wasmtime serve --reuse or wasmtime serve --no-reuse

Benefit

Amortize costly runtime or application initialization across multiple requests.

Implementation

Not sure!

Alternatives

Use a different wasi-http host other than wasmtime in development that supports instance reuse?

[bjorn3]

Amortize costly runtime or application initialization across multiple requests.

Wizer is intended to help with that. It takes a wasm module, runs the initialization code in it and then produces a new wasm module with all initialization already done. It does require some minor changes to your own code to indicate when initialization is done though.

[alexcrichton]

I've got an old branch of mine that I was experimenting with for this which I just rebased to dust it off a bit. There's been some discussion amongst folks historically about this and one of the concerns historically has been about certain instances either requiring reuse or explicitly not supporting reuse and how to surface that in the component itself. Those discussions never reached a firm conclusion one way or another though which is why it's never landed as a feature yet. I'll note that I'm not intimately familiar with the concerns here and I'm mostly a proxy for other (e.g. @lukewagner I think knows much more about the concerns both ways here).

Otherwise I'll definitely echo @bjorn3's idea of wizer. I suspect @ydnar you're already familiar with it but for anyone else who happens upon this in the future we've done a lot of work in Wasmtime to make instantiation as fast as possible through tools like wizer, APIs like InstancePre, and allocators like the pooling allocator in Wasmtime. That often can make instantiation fast enough for scenarios that instance reuse isn't required.

I'd also still acknowledge though that even with all the tools in the toolbox I think there's still real situations where instance-per-request isn't fast enough. Personally I think we should support a flag like this on the command line (the default being perhaps a separate question).

[ydnar]

Thanks!

Wizer works to a point, but cannot help amortize long-lived connections to upstream servers (like databases).

Echoing some live conversations with @alexcrichton and @lukewagner: I think the design of wasi:http/proxy implies reuse, but does not mandate it. A host can choose to not reuse, and a guest can exit to indicate it doesn't want to be reused.

My specific use case: I'm working on a talk for WasmCon with @mossaka, demoing a Go implementation of wasi-http, and would love to show off the new go:wasmexport feature that integrates with goroutines and the runtime.

[lukewagner]

FWIW, a summary of the idea proposed in component-model/#307 is:

• have wasmtime serve default to instance-per-request, but offer a flag to enable instance reuse
• define a new (non-custom) section in component binaries containing a "reuse hint"
• if wasmtime serve instantiates a component that transitively contains the "reuse hint" without the instance-reuse-flag being set, an informative console message is issued

This would seem to me to balance the many subtle competing concerns.

[ydnar]

define a new (non-custom) section in component binaries containing a "reuse hint"

A couple challenges I see with this approach:

• It changes the binary format, requiring tooling changes.
• It requires tooling to thread intent ("I want this component to [not] be reused") from code, pragma, or flags, through to the resulting binary. Where does a component or module author express this intent?
• It is static—versus the component itself indicating, dynamically, at runtime, that it should or should not be reused.

I’d favor an alternative, where:

• Instance reuse is expected and allowed, not not required.
• Hosts can choose how much reuse to offer, from zero to unbounded. This maps to existing ephemeral/"serverless" deployment models.
• Instances can choose to not be reused by exiting e.g. calling something like wasi:cli/exit.
• Instances can indicate they wish to be reused [indefinitely], but the host does not have to honor it.

[lukewagner]

Hrm, I'm not sure if what you're proposing is any different...

• Instance reuse is expected and allowed, not not required.
• Hosts can choose how much reuse to offer, from zero to unbounded. This maps to existing ephemeral/"serverless" deployment models.

These are already the case (both in the component model in general and explicitly in WASI HTTP's doc comments).

• Instances can choose to not be reused by exiting e.g. calling something like wasi:cli/exit.

This too is also already possible. (I'll point out that, since wasi:cli/exit is essentially just a trap without the negative connotation, this is an anti-composable thing for a component to do, since it tears down the whole component DAG including any client components that probably didn't want to die just because they called an imported utility function. There are better options, but I think they're not relevant here.)

• Instances can indicate they wish to be reused [indefinitely], but the host does not have to honor it.

This is what the "reuse hint" is. I don't know if you're imagining some more dynamic indication (calling an imported function or something), but that's no less work (perhaps more) and has less up-front information that can be used by the runtime/platform to provide useful diagnostics to the developer. I also don't know if you're worrying about what's expressible in the core webassembly build target, but we can always express the reuse hint as some symbolic core wasm import or export that wasm-tools component new (or core runtimes) understand the meaning of.

Lastly, I don't know if you're proposing to change the default behavior of wasmtime serve (when no flags are specified), but I think a number of folks have strong opinions on that (since it preemptively gives up temporal isolation), so if you are I'd suggest separating out that discussion from the more mechanistic discussions of expressing intent here.

[ydnar]

Thanks for breaking down the overlapping problem spaces here!

• I think Wasmtime should, by default, reuse instances. I think it’s more straightforward to define temporal isolation as a subset of behavior, rather than reuse as a superset.
• As for a component expressing whether it wants to be reused: I’d favor dynamic over static (code instead of configuration). If could be as simple as a well-known or "core" export that returns a bool or enum value.