Consider defining an "instance reuse hint"

[lukewagner]

There's an interesting question and discussion in wasi-http/#95 that, by the end, doesn't feel specific to "HTTP" at all and thus perhaps deserving of being addressed more generally in the Component Model.

So the basic question is: when a host is given a component to run, does the host reuse the component instance between export calls (and, if so, to what degree?) or does the host create a fresh instance every time. In general, there are pragmatic benefits to using a fresh instance each time (mitigating exploits, clearing out leaks, less non-determinism) which takes advantage of wasm's potentially very-low startup cost. However, there are many valid reasons why a component can have an expensive-enough initialization (calling non-deterministic imports and thus not wizer-able) that this instance-per-export-call default will lead to unacceptable performance. If some hosts reuse instances and others don't, then the resulting performance difference may be significant enough to be a real portability problem. As with core wasm, while it's hard to explicitly specify a "cost model", it ends up being an important implicit part of the design, so I think it's worth thinking through what we want to actually happen and what to tell producer toolchains and runtimes.

First, to enumerate some "can't we just"s that are tempting but I don't think fully address the problem:

• Do nothing. This will initially lead to unportability and then, once the pain is felt, lead to...
• Say that the default is to reuse instances, but that hosts are free to isolate more-aggressively if they want. Over time, hosts will be pressured to adopt a default-reuse stance (anything else will occasionally surprise their users and make them look bad compared to other hosts that default-reuse) with the net-effect being to flip the default away from the pragmatically-safer-and-more-resilient option (and defaults matter).
• Say that a component that needs instance reuse shall export a special "init" function (with a standardized name). E.g., in wasi-http/#95, I initially suggested a wasi:io/slow-init interface. This approach really wants the init function to be called transitively/automatically (just like the start function), making it more "magical" than most WASI interfaces to the toolchain and raising the question of why not build it into the C-M. More subtly but problematically: in general, adding an export to a component is expected to be a backwards-compatible action (the new component is a subtype of the old component's type, after all). But with this magic init function, that expectation breaks: if some component (in your transitive dependency DAG) adds an init function, everyone now needs to recompile, despite there being not obvious type error (you just need a special case the magic "init" interface everywhere this problem might arise).
• Add a runonce function attribute that components can use to signal that a component instance must not be reused. Just because a component instance may be reused doesn't mean it should be reused. Moreover, runonce is rather client-unfriendly: when JS or Python or Rust or any language with module-like bindings imports a component, the default expectation is "I can call the imported functions N times"; runonce would be a totally foreign and annoying constraint. Thus, components wouldn't want to set it, but if setting it is how you get nice temporal isolation, you're encouraged to set it unnecessarily.

Given all that, the best (least-bad) option seems to be the following:

• Hosts default to creating a fresh instance per export call when given the choice, but offer an explicit configuration option to enable instance reuse (along with additional policy options to configure autoscaling, etc).
  • As a notable exception, when native languages runtimes (like JS or Python) import components as-if they were singleton instances (which is the default assumption in most languages), instances would need to be reused (the host doesn't really have a choice).
• The C-M defines a new "instance reuse hint" custom section (analogous to branch hinting or compilation hints).
  • Because it's a custom section, it is not allowed to affect runtime semantics (b/c stripping a custom section should never change runtime behavior).
• If a host is given a component to run that transitively contains the instance reuse hint and the host is not configured to reuse instances, the host is expected to inform the developer (via some sort of warning message) that there may be a perf problem and how to explicitly enable instance reuse.

So yeah, the proposed solution is "a hint", which never feels like winning, but given all the constraints, it feels like the least-bad option. What I like about this approach is that:

• There are natural incentives for component authors to not over- or under-set the instance reuse hint.
• Changes in transitive dependencies (e.g., on version update) don't trigger implicit massive semantic changes while at the same time all components are not forced to manually propagate the instance reuse hint of all their transitive dependencies.
• The developer feels appropriately involved at the right point to make an important decision where we can't simply eliminate one option.
• It makes clear that the host can ultimately do what it wants (reuse instances or not).

Sorry for the long comment; happy to hear more thoughts on this!

[yoshuawuyts]

Being able to reuse types between invocations makes a lot of sense to me. At a previous employer a sibling team was building an integration with AWS' serverless offering, and providing a way to keep persistent connections between request invocations was definitely a challenge. Providing a canonical way to do this not just in the Proxy World, but in the C-M in general makes sense to me.

Sketching a high-level Rust projection

I like to think about through ideas like these by sketching out what the end-user experience for this could look like. Here is how I think we could expose this concept to the Rust projection of the Proxy World in a reasonably ergonomic way:

/// A shared "state" object which is
/// reused between requests.
struct ReusableState(MyDatabaseClient)

/// Implement the WASI constructor for the shared
/// state object. This type will be constructed
/// once before we're ready to handle requests.
impl wasi::init::PreInit for ReusableState {
    async fn construct() -> wasi::init::Result<Self> {
        let db_client = MyDatabaseClient::connect(..).await?;
        Ok(Self(db_client))
    }
}

/// The main entry point to an HTTP proxy world. It
/// takes an owned request and returns an owned
/// response. However crucially: it also takes a
/// shared reference to a pre-initialized state.
#[wasi::http::main]
async fn main(
    req: Request,
    state: &mut ReusableState
) -> wasi::http::Result<Response>> { .. }

Evaluating the instance reuse hint proposal

The way I'm understanding the proposed rules is that what we're roughly saying is that we can tell runtimes they should probably reuse instances, but we can't require them to. The semantics we want is to have a part of our application which is reused between instances, but another part which is ephemeral. I think the "export an init function" option very directly described this use case; but for the stated reasons it's unlikely to work well.

Looking at this projection, I believe the "init function" semantics should still largely be feasible - with the main change in behavior being that we don't actually guarantee that pre-initialization will happen. We can at best hint it will. And I think that's probably fine actually; for the purpose of this sketch there would be no meaningful difference. Did I understand the proposal correctly?

Something I'm unclear about, but think might be relevant here is: how would the hints proposal here grapple with slow initialization? Semantically we want to ensure that we are able to (succesfully) connect to the database before we're ready to accept a request. With the init function it seems clear to me how that would work. Could we do the same thing with just the hints?

[fitzgen]

One of the Web's worklet APIs had this same issue of reuse and the user assumptions that it can build in (eg makes things hard to introduce parallelism to, since users may assume that their main thread instance is being reused). Their solution was to guarantee that at least two instances are created and are either switched between on each task or a worklet is randomly chosen for each task (I can't remember which).

I don't think we should necessarily do the exact same thing, but I think it is worth evaluating and looking to for inspiration.

Could we, for example, spec Bernoulli sampling on each request (or whatever new chunk of work for the instance to process) with some low-ish probability where if the sample returns true, then you must re-instantiate? As long as the probability isn't so low that it is effectively P=0, that would allow reuse but without letting users to start assuming/relying on the shared global state inside the instance. So maybe we would need to spec a minimum P=0.1 or something.

[lukewagner]

@yoshuawuyts The Rust code you wrote makes sense, but one important thing to note is that, even without the hint set, in Preview 2 and beyond, the client of a component is always allowed (according to C-M validation and runtime rules) to call an imported- or child-instance's exports more than once on the same instance. Thus, e.g., a well-behaved wasi:cli/command@0.2.0 component can have its run function called multiple times; the only question is: what does the component implementation do about it? Options include:

• ignore the situation, which likely leads to weird crashes or corruption
• trap, because it's better than the first option
• internally reset the component-instances' state and execute main() again, as-if it was the first time
• actually reuse the instance state (as in your example), assuming the programmer expects it (depending on the languages' idioms and conventions)

So yes, as you said, we can't force the client of a component to reuse instances (at the C-M spec level), but we also can't force the client not to reuse instances. Components are simply dylib-/reactor-/module-like in nature, so it's up to producer toolchains to decide what to do about it (ideally, not option 1).

Something I'm unclear about, but think might be relevant here is: how would the hints proposal here grapple with slow initialization? Semantically we want to ensure that we are able to (succesfully) connect to the database before we're ready to accept a request. With the init function it seems clear to me how that would work. Could we do the same thing with just the hints?

I think the answer here is that, since the start functions inside a component are necessarily run before the first export is called and are fully able to call imports, the DB setup code should be called by the start function. Hosts will naturally exhibit a variety of behaviors for how eagerly/lazily the start functions are called (considering "provisioned concurrency"-vs-"scale to zero", auto-scaling and spikey-workloads), but an optimizing host would be well situated to pre-execute start.

@fitzgen That's a cool idea and makes a bunch of sense in the concrete setting of a browser. Unfortunately, given how open-ended the execution environment of components are, I don't know if we can specify that in a WASI setting. Also, there's a bit of a tragedy-of-the-commons situation where production platforms are incentivized to minimize likelihood of bustage, which may even be the right thing to do by their customers; production is the worst place to activate a latent bug. Also, plenty of valid component embeddings will have their own rather-specific idea of when it makes sense to reuse or not reuse instances. I think our best lever here is the default behavior of what folks will use for local testing, e.g., wasmtime serve. Also, for this and many other reasons, I think Wasmtime should have a --testing flag that activates a "chaos mode" that tries to hit all the rare corner cases in instance reuse, async I/O scheduling, thread scheduling, etc. If it provides enough value in catching bugs early, it could become the default way everyone tests locally, which is a much better time to catch bugs.