New Tier-3 target: `wasm32-wasi-preview2`

[alexcrichton]

Proposal

Note: This proposal is intended to augment a previous MCP for the same target string and update its definition. This MCP is written from the perspective of the compiler today where wasm32-wasi-preview2 is not implemented. This will reiterate some design points of the prior MCP (also because I did not find that MCP and forgot about it before I wrote this all up). More information about the relation to that MCP is available in a section at the end.

This is a proposal to add a new Tier 3 target to the Rust compiler named wasm32-wasi-preview2. This new target will be unlike preexisting WebAssembly targets in two major ways:

• The default binary output of this target would be a WebAssembly component, not a core wasm module.

• The standard library would not use functions from the wasi_snapshot_preview1 namespace, commonly referred to as "WASI Preview1". Instead it would use functions from "[WASI Preview2]". These functions are defined at the component abstraction layer, not core wasm.

This target is not to be confused with wasm32-wasi, the currently existing WASI target in the Rust compiler. That target is in the process of being renamed to wasm32-wasi-preview1 to precisely make way for this target. The wasm32-wasi-preview1 target, and current wasm32-wasi target, are effectively at a dead-end in the design space of WASI itself, and the preview2 iteration represents the next phase of development of WASI itself.

Note that this target is also not to be confused with the wasm32-wasi-preview1-threads target. The Component Model does not have support for threads at this time and that's not expected to be supported until the time scale of years in the future. Eventually there will likely be a wasm32-wasi-preview2-threads target but that's not being proposed at this time.

Background: Component Model

To help better understand this MCP and what it means for the Rust compiler I wanted to also provide some background on the motivations for what's going on here. The first bit to clarify is the component model itself. The Component Model is a WebAssembly proposal currently at "Phase 1" which is the introduction phase of features into WebAssembly. Despite this early phase of the Component Model it's been in development for nearly 5 years now and has had quite a bit of progress in the meantime. The wasm-bindgen project in Rust, for example, is a major source of design for the component model historically.

The Component Model is a specification that is layered on top of "core WebAssembly" which is what "wasm" typically refers to today. The component model defines a "component" which internally contains core wasm modules. One major feature of a component is that describes imports and exports with rich types such as string or tuple<u32, char, string>. This is in contrast to core WebAssembly where all functions take integral/floating point types. One purpose of the component model is to provide easier integration of WebAssembly both on the web and off the web. For example wasm-bindgen also supports richer types than i32 but it requires JS and the web, whereas components are runnable off-the-web without supporting JS (and on the web too with a polyfill).

The component model is reaching a milestone of stability at the end of this year with the release of WASI Preview 2. The component model's binary format has been stable for a number of months now and the ecosystem around components is expected to provide longer-term stability for the current binary format specification. This is not an ironclad guarantee of stability and it is guaranteed to change in the future, but that timescale is in years.

Background: WASI

The second major feature of this proposed wasm32-wasi-preview2 target is that it will use the next iteration of WASI for the implementation of the standard library. Currently the wasm32-wasi target uses APIs from the wasi_snapshot_preview1 module. This namespace is defined by the WASI subgroup of the WebAssembly Community Group, the main standardization body of the WebAssembly specification. The wasi_snapshot_preview1 namespace is the original iteration of design for WASI and was intended to be a "get the foot in the door" sort of phase of design where it was the best that could be done at the time.

Since its introduction the wasi_snapshot_preview1 has served the WASI subgroup well in providing feedback for future iteration and design. Namely there are fundamental design aspects of the wasi_snapshot_preview1 module which required changing to satisfy design requirements moving forward. One major design requirement is that WASI is today now defined in terms of the component model, not core WebAssembly. This means that wasi_snapshot_preview1 is incompatible and has no "easy" translation into the component model since its whole purpose was to be defined at the core wasm layer.

This redefinition in terms of the component model means that APIs from wasi_snapshot_preview1 have largely all changed. All functionality is preserved "in spirit" in the preview2 snapshot of WASI, however. For example programs can still get environment variables, print to stdout, open files, etc. The precise way that all these APIs works now is defined via two pieces:

1. All WASI APIs are defined with WIT, an IDL suitable for code generation in many languages and environments. WIT functions are all defined in terms of components so there is a clear mapping to the component model.

2. Component model functions can be "lowered" into core WebAssembly functions using the Canonical ABI of the component model. This provides a clear definition for what it means to communicate a string to a core wasm module for example.

This means that WASI is a collection of *.wit files (this is the current snapshot in Wasmtime for example). These WIT files describe interfaces, functions, and types which can all be imported into WebAssembly core wasm modules. From these WIT files a bindings generator such as wit-bindgen is used to generate Rust functions to interface with these WIT interfaces.

Background: Why Now?

The Component Model and WASI itself are in development. Neither provides an ironclad guarantee of stability, and in fact both are guaranteed to change over time instead. Given this nature, why would Rust want to add a wasm32-wasi-preview2 target at this time?

In some sense this is a chicken-and-egg problem. For the further development of the Component Model and WASI itself developer feedback is required. We would be nowhere near where we are today without the Rust wasm32-wasi for example. If stability is required before adding a target to the Rust compiler, however, then there's no great way to break this cycle and bootstrap somehow. This is more-or-less a way of saying that a new target in Rust is desired as a means to develop these proposals and move them towards stability since without a Rust target stability is unlikely to be achieved due to lack of developer feedback.

Despite this though we (those working on the component model) didn't try to get a Rust target from day 1 for the component model because it was understood that the churn would be too high. The feeling though is that now is where the component model and WASI have reached a level of stability that it's now suitable to have a Rust target. While the Rust target isn't be guaranteed to be present and unchanging until the end of time there are still "more stable" foundations on which to build this target:

• The binary format of components has been stable for some time now and is expected to be supported at this version for quite some time going forward. No known major changes are left and any future iterations to the binary format will be required to support everything in the binary format today, meaning that any changes should in theory be cosmetic. This means that the base level of the output should be stable enough in theory for a Rust target.

• WASI itself is expected to enter a period of stability. Lots of churn has happened up to the release of preview2, but the upcoming version release of 0.2.0 is expected to be a milestone in the development of WASI. APIs will not break or change throughout the development of 0.2.x which is expected to have a lifetime in the scale of years. Iteration, however, is expected. It is planned to have an 0.2.1 release with added functions/interfaces for example in the future (no concrete timeline at this time).

These combined together mean that the wasm32-wasi-preview2 target is expected to have a good degree, albeit not ironclad, sense of stability moving forward. Major breaking changes are expected to be evaluated as necessary against the entire component ecosystem and a suitable proposal would be brought forward for both Rust and other languages. An "easy way out" would be deprecation of wasm32-wasi-preview2 and the introduction of wasm32-wasi-preview3 for example. This is not guaranteed to happen, though, and it depends on how the development of preview2 goes.

To reiterate, the trajectory of the wasm32-wasi-preview2 target is uncertain at this time. While it's expected to have stability on the time scale of a year or so further out than that is unknown. It highly depends on how the component ecosystem shapes up, developer feedback from integrating this target with Rust, etc. The introduction of the target, however, is seen as crucial to providing this feedback.

What's in the Target: libstd

Ok with all that written down my hope is that the existence of this target is sufficiently motivated for inclusion into the Rust compiler and standard library. Here I'd like to discuss some more details about what this change might concretely look like.

The first, and most desirable, change would be to the standard library itself. Currently the standard library has a std::sys::wasi module which implements Rust's platform-specific APIs in terms of WASI APIs. The standard library additionally interacts with wasi-libc to ensure that Rust and C sources can be linked into the same WebAssembly binary with interoperability going through the wasi-libc library. In this new target the std::sys::wasi submodule will be renamed as std::sys::wasi_preview1 and then reimplemented as std::sys::wasi_preview2. This could be configured with target_vendor = "preview2" or target_os = "wasi_preview2" for example (TBD).

The new sys submodule would not use the wasi crate from crates.io as is done today with the wasm32-wasi target because that crate uses wasi_snapshot_preview1. Instead functions will be generated through wit-bindgen. To have the least impact on libstd's build process the generated files will be checked into the source of the standard library. Hand-written documentation will indicate how the files can be regenerated if need be.

Note, however, that not all APIs may be implemented in terms of the raw WASI interfaces. The wasm32-wasi-preview2 target will still use wasi-libc and may use some APIs from there. For example WASI preview2 has support for sockets but they don't look like POSIX sockets. The wasi-libc library will implement a POSIX-like API in terms of WASI preview2. This means that sockets in Rust can either use wasi-libc's APIs or WASI preview2 APIs directly. Precisely how the Rust standard library chooses to implement primitives is not something this proposal wants to set in stone. Some primitives might go one way while some may go the other way. It's intended that the best means of implementation is iterated on over time internally in the standard library. A major factor in deciding this will be figuring out what the AsRaw* traits will look like for this target. WASI has no concept of a "file descriptor" but at the same time the representation of sockets is quite different than POSIX for example, meaning that there's not an obvious 1:1 mapping from standard library primitives to WASI primitives.

Overall the exposed abstraction layer is something that this proposal does not intend to specify at this time but wants to highlight will be a point of development for this new target. Integration with ecosystem crates doing async I/O, for example, is expected to help guide this design and inform what the best choices are here.

What's in the Target: rustc

The final piece I'd like to talk about in this MCP are the changes to rustc required for this target. The easy part of these changes is the introduction of a wasm32-wasi-preview2 target specification in rustc_target. This won't be all that different than the other WebAssembly targets with one major exception. Currently all WebAssembly targets that rustc supports emit core WebAssembly modules but I've indicated that this new target would emit a WebAssembly component instead. Here I'd like to explain how that is envisioned to work.

As background all emission of core WebAssembly modules is done through wasm-ld today. The wasm-ld binary is provided by LLVM as a flavor of LLD, LLVM's linker. The wasm-ld binary does not have support for components and it is not planned to add support to emit components at this time. The development of the component model to-date has worked with the wasm32-wasi target for example which emits a core WebAssembly binary using wasi_snapshot_preview1. This core wasm binary is then transformed into a component using two pieces:

• The wit-component project is primarily used to transform a core wasm module into a component. This uses type information embedded by wit-bindgen into wasm custom sections to generate the final component. The custom sections are stripped from the input module and the final component is created.

• The wasi_snapshot_preview1 imports are translated to preview2 through the use of an "adapter". This adapter exports symbols matching those in wasi_snapshot_preview1 and then internally translates to preview2. This adapter is itself a core wasm module and is present in the final output component.

Given all this background, the purpose of wasm32-wasi-preview2 is to bundle all this together for users. The compiler will effectively run wit-component for each compilation in addition to using an adapter if necessary. The current vision for this is to decouple all of this from rustc itself by using a custom linker binary distributed with the compiler.

Today the rust-lld binary is built by the Rust project's CI and distributed for all major targets. This new linker, I'll call it wasm-component-ld for now, would also be built by the Rust project's CI and distributed for all platforms. This means that the wasm32-wasi-preview2 target would be composed of both this linker in addition to the standard library itself. This separation achieves a few goals:

1. Primarily the stability of the wasm-component-ld tool isn't tied to rustc itself. All the code will live in its own binary off to the side for exclusive use for this one target.
2. This additionally provides the flexibility to skip this process altogether. For example a linker of wasm-ld could be specified to emit a core wasm module instead of a component in a non-default fashion. This can be useful for some component building processes.
3. This eventually provides the opportunity for this project to be developed externally to rustc itself for use with other languages too.
4. Coupling the distribution of the linker and the standard library together enables customizing the adapter used for libstd's purposes. For example if the standard library nor wasi-libc use wasi_snapshot_preview1 then there's no need for an adapter. This is expected to take some time to develop though so for the early lifetime of the wasm32-wasi-preview2 target it's expected an adapter will be required. As the target evolves the linker can transparently evolve as well.

Despite the intention to eventually bundle this binary my intention would be, for the initial implementation, to not bundle anything. This would mean that the wasm32-wasi-preview2 target would, by default, not actually work for anyone since it would invoke a nonexistent linker. This should help provide a bit of an early speedbump to enable changes if necessary to this linker scheme. The intention is that this linker would be developed externally to get to a "proof of concept" stage and then if and when wasm32-wasi-preview2 approaches Tier 2 in Rust it would move to being distributed at that time.

What's in the target: Changes from Today

Adding a new target is a good opportunity to fix mistakes and/or issues with the wasm32-wasi target that this is in the long-term going to replace. While these issues can theoretically be fixed on the existing target it typically requires a good deal of effort to keep things working with smoother transition periods. Some examples of issues which can be fixed/changed with this new target are:

• All code will be compiled with an -fPIC equivalent by default. This enables much more easily producing a "shared everything" library in wasm for Emscripten-style linking against other libraries in the same component. This is already being done for a Rust extension to Python compiled to wasm but requires -Zbuild-std for now. The impact of this is expected to be minimal if at all measurable.
• If needed crate-type outputs could be adjusted. Currently the "bin" crate type generates a "command" with a main function which is generally agreed to be ok. The cdylib crate type, however, is a bit odder in wasm where currently it generates a wasm file without an entry point (sometimes referred to as a "reactor"). There's also a use case for *.so equivalents for the Emscripten-style dynamic linking above. How exactly this is mapped to Rust crate types is a bit unclear, but this new target provides an opportunity to adjust these as appropriate to get things working.

If others have grievances or things they'd like to adjust in WASI Rust targets this would be a great time to consider them as well (especially before Tier 2 is applied for)

Benefits - Future and Immediate

I realize that this is a nontrivially-sized proposal for the Rust project. Here I'd like to explain some benefits which will result from this work to help further motivate why this should be done.

Perhaps the largest reason is that the current existing wasm32-wasi (to be renamed to wasm32-wasi-preview1) is at a "dead end" from a spec-design perspective. It's expected that this target will still be widely used for quite some time while components fully mature, but from a standards perspective no new functions are being added. For example std::net::UdpSocket doesn't work on WASI as there is no specification of APIs to use, nor will there ever be from the WASI subgroup. By switching the design of the Rust target to the preview2 snapshot of WASI it moves the target onto a trajectory to follow the future development of the WASI standard. For example UDP APIs are specified in the preview2 snapshot of WASI, but the standard library cannot make use of them on the wasm32-wasi target, only on this new wasm32-wasi-preview2 target which targets components.

This new target additionally assists in getting out of the "rut" of adding new features to the standard library for newer WASI APIs. Currently it's not clear how to integrate the standard library with preview2 on the preexisting target. Additionally there's quite a lot of work to do switching everything over to preview2 from preview1. This target is designed to provide a ramp by which incremental steps can be made toward the eventual goal of implementing everything in terms of preview2. Initially the standard library will look exactly the same for preview1 and preview2, but with a new target that uses components as output it's possible to implement new APIs in terms of preview2 (e.g. UDP sockets). The high-level specification of the output (a component as opposed to a core wasm module) provides a good deal of flexibility for the standard library to evolve over time as opposed to being required to be perfect and final from day 1.

Finally this is going to be a crucial step for the evolution and standardization of the component model itself. This greatly lowers the barrier to entry to writing Rust programs to generate components and provide feedback for implementations. This will enable seeding preview2 support in the ecosystem as well, for example one day tokio could work on preview2 in addition to reqwest using APIs from preview2. None of that can easily happen until a Rust target is added first.

Relation to the prior MCP for wasm32-wasi-preview2

The wasm32-wasi-preview2 target has had a prior MCP to add it as a Tier 2 target. I unfortunately forgot about that and additionally didn't search well enough to find it, so here I'd like to address the differences and why I think a new MCP is necessary.

This proposal has two major changes relative to the prior MCP, both of which build on each other:

1. The binary format is proposed here to be a component, not a core wasm module. The primary motivation for this is to best align with the Component Model itself and how WASI is specified. This additionally, however, enables a more incremental approach to standing up this target rather than requiring it to be "finished" from day one. For example this won't require transitioning away from using preview1 as an internal implementation detail via the usage of adapters mentioned above.
2. This change in the binary format is novel for rustc and additionally requires a new linker. Given this substantial change relative to the prior target it's expected that some iteration will be required to "get this right", meaning that this proposal starts at Tier 3 instead of Tier 2. This is hoped to be able to buy some runway in terms of breaking changes in the linker and figuring out and fixing bugs before everything is finalized before Tier 2.

Otherwise though the spirit of the prior FCP is preserved here, and it likely turns out that much of the words in this MCP aren't necessary to be restated as it was agreed upon for the first MCP.

Next Steps

If this proposal is accepted then I wanted to also outline a few steps of what would happen next with this target. The end goal would be to promote this target to Tier 2 to enable rustup target add wasm32-wasi-preview2 to be the primary end-user experience. Before this, however, the steps might look like:

• Add wasm32-wasi-preview2 to the compiler
• Start a wasm-component-ld project which proxies to wasm-ld (e.g. rust-lld) and then runs wit-component
  • TBD: CLI flag syntax for configuring wasm-component-ld vs wasm-ld
  • TBD: CLI flags passed from rustc to wasm-component-ld, e.g. does it pretend to "just" be wasm-ld?
• Determine how to disambiguate wasm32-wasi and wasm32-wasi-preview2 in Rust code (e.g. cfg(target_os) or cfg(target_vendor))
• Get the standard library compiling for wasm32-wasi-preview2, e.g. configuring cfg(target_os = "wasi") to additionally work with this new target. For now this means wasm32-wasi-preview2's intermediate core wasm binary would still reference wasi_snapshot_preview1
• Implement enough of wasm-component-ld with a bundled adapter to generate a component.
• Test that everything works on nightly with cargo build -Zbuild-std plus environment configuration (e.g. wasm-component-ld is in $PATH). At this point there's feature parity with wasm32-wasi plus a manual wit-component run.
• Implement full support for std::net build on WASI preview2
  • Option A: implement this directly in Rust using WASI APIs directly as-generated by wit-bindgen
  • Option B: implement a POSIX API in wasi-libc and use that in libstd
• Start integrating the wasm32-wasi-preview2 target into community crates as appropriate (e.g. libc)
• Flesh out all of the above into a more "final" state
• Apply for Tier 2 status which would entail (in addition to Rust's preexisting requirements)
  • Working components with -Zbuild-std for a variety of applications/embeddings.
  • Full support for std::net implemented in the standard library.
  • Full support for std::fs and std::env as they exist today.
  • Integration with ecosystem crates such as getrandom and libc, perhaps even mio as a stretch goal.
  • Stability story for wasm-component-ld and its arguments and inputs finalized.
  • Agreed-upon distribution plan for wasm-component-ld alongside rust-lld with the Rust compiler distribution.

At this point, which is expected to be ideally no more than a year from Tier 3 acceptance, the intent is to apply for Tier 2 status for distribution through rustup to get more feedback for the component model an WASI.

Mentors or Reviewers

I plan on personally being on-the-hook for any reviews to the capacity that I'm qualified to perform such a review. I won't be able to review integration with the rest of the standard library since that falls under the libs implementation team, but I can review any WASI-specific parts, however.

Process

The main points of the Major Change Process are as follows:

• [x] File an issue describing the proposal.
• [ ] A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• [ ] Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[yoshuawuyts]

If I'm not mistaken this might be a duplicate of the already-accepted MCP in https://github.com/rust-lang/compiler-team/issues/594?

[alexcrichton]

Oh dear I'm so sorry for missing that when searching and additionally forgetting about that!

After some dicussion with Yosh we agree that this proposal is different enough it's likely best to get agreement again given the change in direction. I've added a "Note" to the top of this proposal in addition to a new "Relation to the prior MCP" section.

[ranile]

There's one concern I have with this approach: do we just keep adding new targets as new WASI previews come out? I assume we would want to stabilize the targets so people can use them without nightly. What happens when preview3 come out? Will we add a -preview3 target?

I feel like a better solution is to update wasm32-wasi such that it works with preview 2. It can emit a core wasm module which can then be converted to a component. If need be, there can be a target/compiler flag that also emits the binaries compatible with component modal. The big argument against this is breaking all preview1 (current) users. How badly will we break them, if such a change were to be made? I think it's safe to assume that moving forward, the runtimes will all be abandoning preview1 so the need for it isn't much. We can perhaps keep a way to opt into preview1 output for a certain amount of time, maybe?

[Noratrieb]

Technical discussion like this should go to the Zulip thread mentioned above.

[ranile]

I raised the concern here because the issue said:

Use this issue to leave procedural comments, such as [...] raising a concern that you would like to be addressed.

I'll repost it on Zulip if that's better

[davidtwco]

@rustbot second

This strikes me as reasonable and I'm satisfied that all concerns raised in this thread and in Zulip have been considered.

[apiraino]

@rustbot label -final-comment-period +major-change-accepted