Closed alexcrichton closed 7 months ago
This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.
cc @rust-lang/compiler @rust-lang/compiler-contributors
If I'm not mistaken this might be a duplicate of the already-accepted MCP in https://github.com/rust-lang/compiler-team/issues/594?
Oh dear I'm so sorry for missing that when searching and additionally forgetting about that!
After some dicussion with Yosh we agree that this proposal is different enough it's likely best to get agreement again given the change in direction. I've added a "Note" to the top of this proposal in addition to a new "Relation to the prior MCP" section.
There's one concern I have with this approach: do we just keep adding new targets as new WASI previews come out? I assume we would want to stabilize the targets so people can use them without nightly. What happens when preview3 come out? Will we add a -preview3 target?
I feel like a better solution is to update wasm32-wasi such that it works with preview 2. It can emit a core wasm module which can then be converted to a component. If need be, there can be a target/compiler flag that also emits the binaries compatible with component modal. The big argument against this is breaking all preview1 (current) users. How badly will we break them, if such a change were to be made? I think it's safe to assume that moving forward, the runtimes will all be abandoning preview1 so the need for it isn't much. We can perhaps keep a way to opt into preview1 output for a certain amount of time, maybe?
Technical discussion like this should go to the Zulip thread mentioned above.
I raised the concern here because the issue said:
Use this issue to leave procedural comments, such as [...] raising a concern that you would like to be addressed.
I'll repost it on Zulip if that's better
@rustbot second
This strikes me as reasonable and I'm satisfied that all concerns raised in this thread and in Zulip have been considered.
@rustbot label -final-comment-period +major-change-accepted
Proposal
This is a proposal to add a new Tier 3 target to the Rust compiler named
wasm32-wasi-preview2
. This new target will be unlike preexisting WebAssembly targets in two major ways:The default binary output of this target would be a WebAssembly component, not a core wasm module.
The standard library would not use functions from the
wasi_snapshot_preview1
namespace, commonly referred to as "WASI Preview1". Instead it would use functions from "[WASI Preview2]". These functions are defined at the component abstraction layer, not core wasm.This target is not to be confused with
wasm32-wasi
, the currently existing WASI target in the Rust compiler. That target is in the process of being renamed towasm32-wasi-preview1
to precisely make way for this target. Thewasm32-wasi-preview1
target, and currentwasm32-wasi
target, are effectively at a dead-end in the design space of WASI itself, and the preview2 iteration represents the next phase of development of WASI itself.Note that this target is also not to be confused with the
wasm32-wasi-preview1-threads
target. The Component Model does not have support for threads at this time and that's not expected to be supported until the time scale of years in the future. Eventually there will likely be awasm32-wasi-preview2-threads
target but that's not being proposed at this time.Background: Component Model
To help better understand this MCP and what it means for the Rust compiler I wanted to also provide some background on the motivations for what's going on here. The first bit to clarify is the component model itself. The Component Model is a WebAssembly proposal currently at "Phase 1" which is the introduction phase of features into WebAssembly. Despite this early phase of the Component Model it's been in development for nearly 5 years now and has had quite a bit of progress in the meantime. The
wasm-bindgen
project in Rust, for example, is a major source of design for the component model historically.The Component Model is a specification that is layered on top of "core WebAssembly" which is what "wasm" typically refers to today. The component model defines a "component" which internally contains core wasm modules. One major feature of a component is that describes imports and exports with rich types such as
string
ortuple<u32, char, string>
. This is in contrast to core WebAssembly where all functions take integral/floating point types. One purpose of the component model is to provide easier integration of WebAssembly both on the web and off the web. For examplewasm-bindgen
also supports richer types thani32
but it requires JS and the web, whereas components are runnable off-the-web without supporting JS (and on the web too with a polyfill).The component model is reaching a milestone of stability at the end of this year with the release of WASI Preview 2. The component model's binary format has been stable for a number of months now and the ecosystem around components is expected to provide longer-term stability for the current binary format specification. This is not an ironclad guarantee of stability and it is guaranteed to change in the future, but that timescale is in years.
Background: WASI
The second major feature of this proposed
wasm32-wasi-preview2
target is that it will use the next iteration of WASI for the implementation of the standard library. Currently thewasm32-wasi
target uses APIs from thewasi_snapshot_preview1
module. This namespace is defined by the WASI subgroup of the WebAssembly Community Group, the main standardization body of the WebAssembly specification. Thewasi_snapshot_preview1
namespace is the original iteration of design for WASI and was intended to be a "get the foot in the door" sort of phase of design where it was the best that could be done at the time.Since its introduction the
wasi_snapshot_preview1
has served the WASI subgroup well in providing feedback for future iteration and design. Namely there are fundamental design aspects of thewasi_snapshot_preview1
module which required changing to satisfy design requirements moving forward. One major design requirement is that WASI is today now defined in terms of the component model, not core WebAssembly. This means thatwasi_snapshot_preview1
is incompatible and has no "easy" translation into the component model since its whole purpose was to be defined at the core wasm layer.This redefinition in terms of the component model means that APIs from
wasi_snapshot_preview1
have largely all changed. All functionality is preserved "in spirit" in the preview2 snapshot of WASI, however. For example programs can still get environment variables, print to stdout, open files, etc. The precise way that all these APIs works now is defined via two pieces:All WASI APIs are defined with WIT, an IDL suitable for code generation in many languages and environments. WIT functions are all defined in terms of components so there is a clear mapping to the component model.
Component model functions can be "lowered" into core WebAssembly functions using the Canonical ABI of the component model. This provides a clear definition for what it means to communicate a
string
to a core wasm module for example.This means that WASI is a collection of
*.wit
files (this is the current snapshot in Wasmtime for example). These WIT files describe interfaces, functions, and types which can all be imported into WebAssembly core wasm modules. From these WIT files a bindings generator such aswit-bindgen
is used to generate Rust functions to interface with these WIT interfaces.Background: Why Now?
The Component Model and WASI itself are in development. Neither provides an ironclad guarantee of stability, and in fact both are guaranteed to change over time instead. Given this nature, why would Rust want to add a
wasm32-wasi-preview2
target at this time?In some sense this is a chicken-and-egg problem. For the further development of the Component Model and WASI itself developer feedback is required. We would be nowhere near where we are today without the Rust
wasm32-wasi
for example. If stability is required before adding a target to the Rust compiler, however, then there's no great way to break this cycle and bootstrap somehow. This is more-or-less a way of saying that a new target in Rust is desired as a means to develop these proposals and move them towards stability since without a Rust target stability is unlikely to be achieved due to lack of developer feedback.Despite this though we (those working on the component model) didn't try to get a Rust target from day 1 for the component model because it was understood that the churn would be too high. The feeling though is that now is where the component model and WASI have reached a level of stability that it's now suitable to have a Rust target. While the Rust target isn't be guaranteed to be present and unchanging until the end of time there are still "more stable" foundations on which to build this target:
The binary format of components has been stable for some time now and is expected to be supported at this version for quite some time going forward. No known major changes are left and any future iterations to the binary format will be required to support everything in the binary format today, meaning that any changes should in theory be cosmetic. This means that the base level of the output should be stable enough in theory for a Rust target.
WASI itself is expected to enter a period of stability. Lots of churn has happened up to the release of preview2, but the upcoming version release of 0.2.0 is expected to be a milestone in the development of WASI. APIs will not break or change throughout the development of 0.2.x which is expected to have a lifetime in the scale of years. Iteration, however, is expected. It is planned to have an 0.2.1 release with added functions/interfaces for example in the future (no concrete timeline at this time).
These combined together mean that the
wasm32-wasi-preview2
target is expected to have a good degree, albeit not ironclad, sense of stability moving forward. Major breaking changes are expected to be evaluated as necessary against the entire component ecosystem and a suitable proposal would be brought forward for both Rust and other languages. An "easy way out" would be deprecation ofwasm32-wasi-preview2
and the introduction ofwasm32-wasi-preview3
for example. This is not guaranteed to happen, though, and it depends on how the development of preview2 goes.To reiterate, the trajectory of the
wasm32-wasi-preview2
target is uncertain at this time. While it's expected to have stability on the time scale of a year or so further out than that is unknown. It highly depends on how the component ecosystem shapes up, developer feedback from integrating this target with Rust, etc. The introduction of the target, however, is seen as crucial to providing this feedback.What's in the Target: libstd
Ok with all that written down my hope is that the existence of this target is sufficiently motivated for inclusion into the Rust compiler and standard library. Here I'd like to discuss some more details about what this change might concretely look like.
The first, and most desirable, change would be to the standard library itself. Currently the standard library has a
std::sys::wasi
module which implements Rust's platform-specific APIs in terms of WASI APIs. The standard library additionally interacts with wasi-libc to ensure that Rust and C sources can be linked into the same WebAssembly binary with interoperability going through the wasi-libc library. In this new target thestd::sys::wasi
submodule will be renamed asstd::sys::wasi_preview1
and then reimplemented asstd::sys::wasi_preview2
. This could be configured withtarget_vendor = "preview2"
ortarget_os = "wasi_preview2"
for example (TBD).The new
sys
submodule would not use thewasi
crate from crates.io as is done today with thewasm32-wasi
target because that crate useswasi_snapshot_preview1
. Instead functions will be generated throughwit-bindgen
. To have the least impact on libstd's build process the generated files will be checked into the source of the standard library. Hand-written documentation will indicate how the files can be regenerated if need be.Note, however, that not all APIs may be implemented in terms of the raw WASI interfaces. The
wasm32-wasi-preview2
target will still usewasi-libc
and may use some APIs from there. For example WASI preview2 has support for sockets but they don't look like POSIX sockets. Thewasi-libc
library will implement a POSIX-like API in terms of WASI preview2. This means that sockets in Rust can either usewasi-libc
's APIs or WASI preview2 APIs directly. Precisely how the Rust standard library chooses to implement primitives is not something this proposal wants to set in stone. Some primitives might go one way while some may go the other way. It's intended that the best means of implementation is iterated on over time internally in the standard library. A major factor in deciding this will be figuring out what theAsRaw*
traits will look like for this target. WASI has no concept of a "file descriptor" but at the same time the representation of sockets is quite different than POSIX for example, meaning that there's not an obvious 1:1 mapping from standard library primitives to WASI primitives.Overall the exposed abstraction layer is something that this proposal does not intend to specify at this time but wants to highlight will be a point of development for this new target. Integration with ecosystem crates doing async I/O, for example, is expected to help guide this design and inform what the best choices are here.
What's in the Target: rustc
The final piece I'd like to talk about in this MCP are the changes to rustc required for this target. The easy part of these changes is the introduction of a
wasm32-wasi-preview2
target specification inrustc_target
. This won't be all that different than the other WebAssembly targets with one major exception. Currently all WebAssembly targets that rustc supports emit core WebAssembly modules but I've indicated that this new target would emit a WebAssembly component instead. Here I'd like to explain how that is envisioned to work.As background all emission of core WebAssembly modules is done through
wasm-ld
today. Thewasm-ld
binary is provided by LLVM as a flavor of LLD, LLVM's linker. Thewasm-ld
binary does not have support for components and it is not planned to add support to emit components at this time. The development of the component model to-date has worked with thewasm32-wasi
target for example which emits a core WebAssembly binary usingwasi_snapshot_preview1
. This core wasm binary is then transformed into a component using two pieces:The
wit-component
project is primarily used to transform a core wasm module into a component. This uses type information embedded bywit-bindgen
into wasm custom sections to generate the final component. The custom sections are stripped from the input module and the final component is created.The
wasi_snapshot_preview1
imports are translated to preview2 through the use of an "adapter". This adapter exports symbols matching those inwasi_snapshot_preview1
and then internally translates to preview2. This adapter is itself a core wasm module and is present in the final output component.Given all this background, the purpose of
wasm32-wasi-preview2
is to bundle all this together for users. The compiler will effectively runwit-component
for each compilation in addition to using an adapter if necessary. The current vision for this is to decouple all of this from rustc itself by using a custom linker binary distributed with the compiler.Today the
rust-lld
binary is built by the Rust project's CI and distributed for all major targets. This new linker, I'll call itwasm-component-ld
for now, would also be built by the Rust project's CI and distributed for all platforms. This means that thewasm32-wasi-preview2
target would be composed of both this linker in addition to the standard library itself. This separation achieves a few goals:wasm-component-ld
tool isn't tied to rustc itself. All the code will live in its own binary off to the side for exclusive use for this one target.wasm-ld
could be specified to emit a core wasm module instead of a component in a non-default fashion. This can be useful for some component building processes.wasi-libc
usewasi_snapshot_preview1
then there's no need for an adapter. This is expected to take some time to develop though so for the early lifetime of thewasm32-wasi-preview2
target it's expected an adapter will be required. As the target evolves the linker can transparently evolve as well.Despite the intention to eventually bundle this binary my intention would be, for the initial implementation, to not bundle anything. This would mean that the
wasm32-wasi-preview2
target would, by default, not actually work for anyone since it would invoke a nonexistent linker. This should help provide a bit of an early speedbump to enable changes if necessary to this linker scheme. The intention is that this linker would be developed externally to get to a "proof of concept" stage and then if and whenwasm32-wasi-preview2
approaches Tier 2 in Rust it would move to being distributed at that time.What's in the target: Changes from Today
Adding a new target is a good opportunity to fix mistakes and/or issues with the
wasm32-wasi
target that this is in the long-term going to replace. While these issues can theoretically be fixed on the existing target it typically requires a good deal of effort to keep things working with smoother transition periods. Some examples of issues which can be fixed/changed with this new target are:-fPIC
equivalent by default. This enables much more easily producing a "shared everything" library in wasm for Emscripten-style linking against other libraries in the same component. This is already being done for a Rust extension to Python compiled to wasm but requires-Zbuild-std
for now. The impact of this is expected to be minimal if at all measurable.main
function which is generally agreed to be ok. Thecdylib
crate type, however, is a bit odder in wasm where currently it generates a wasm file without an entry point (sometimes referred to as a "reactor"). There's also a use case for*.so
equivalents for the Emscripten-style dynamic linking above. How exactly this is mapped to Rust crate types is a bit unclear, but this new target provides an opportunity to adjust these as appropriate to get things working.If others have grievances or things they'd like to adjust in WASI Rust targets this would be a great time to consider them as well (especially before Tier 2 is applied for)
Benefits - Future and Immediate
I realize that this is a nontrivially-sized proposal for the Rust project. Here I'd like to explain some benefits which will result from this work to help further motivate why this should be done.
Perhaps the largest reason is that the current existing
wasm32-wasi
(to be renamed towasm32-wasi-preview1
) is at a "dead end" from a spec-design perspective. It's expected that this target will still be widely used for quite some time while components fully mature, but from a standards perspective no new functions are being added. For examplestd::net::UdpSocket
doesn't work on WASI as there is no specification of APIs to use, nor will there ever be from the WASI subgroup. By switching the design of the Rust target to the preview2 snapshot of WASI it moves the target onto a trajectory to follow the future development of the WASI standard. For example UDP APIs are specified in the preview2 snapshot of WASI, but the standard library cannot make use of them on thewasm32-wasi
target, only on this newwasm32-wasi-preview2
target which targets components.This new target additionally assists in getting out of the "rut" of adding new features to the standard library for newer WASI APIs. Currently it's not clear how to integrate the standard library with preview2 on the preexisting target. Additionally there's quite a lot of work to do switching everything over to preview2 from preview1. This target is designed to provide a ramp by which incremental steps can be made toward the eventual goal of implementing everything in terms of preview2. Initially the standard library will look exactly the same for preview1 and preview2, but with a new target that uses components as output it's possible to implement new APIs in terms of preview2 (e.g. UDP sockets). The high-level specification of the output (a component as opposed to a core wasm module) provides a good deal of flexibility for the standard library to evolve over time as opposed to being required to be perfect and final from day 1.
Finally this is going to be a crucial step for the evolution and standardization of the component model itself. This greatly lowers the barrier to entry to writing Rust programs to generate components and provide feedback for implementations. This will enable seeding preview2 support in the ecosystem as well, for example one day
tokio
could work on preview2 in addition toreqwest
using APIs from preview2. None of that can easily happen until a Rust target is added first.Relation to the prior MCP for
wasm32-wasi-preview2
The
wasm32-wasi-preview2
target has had a prior MCP to add it as a Tier 2 target. I unfortunately forgot about that and additionally didn't search well enough to find it, so here I'd like to address the differences and why I think a new MCP is necessary.This proposal has two major changes relative to the prior MCP, both of which build on each other:
Otherwise though the spirit of the prior FCP is preserved here, and it likely turns out that much of the words in this MCP aren't necessary to be restated as it was agreed upon for the first MCP.
Next Steps
If this proposal is accepted then I wanted to also outline a few steps of what would happen next with this target. The end goal would be to promote this target to Tier 2 to enable
rustup target add wasm32-wasi-preview2
to be the primary end-user experience. Before this, however, the steps might look like:wasm32-wasi-preview2
to the compilerwasm-component-ld
project which proxies towasm-ld
(e.g.rust-lld
) and then runswit-component
wasm-component-ld
vswasm-ld
wasm-component-ld
, e.g. does it pretend to "just" bewasm-ld
?wasm32-wasi
andwasm32-wasi-preview2
in Rust code (e.g.cfg(target_os)
orcfg(target_vendor)
)wasm32-wasi-preview2
, e.g. configuringcfg(target_os = "wasi")
to additionally work with this new target. For now this meanswasm32-wasi-preview2
's intermediate core wasm binary would still referencewasi_snapshot_preview1
wasm-component-ld
with a bundled adapter to generate a component.cargo build -Zbuild-std
plus environment configuration (e.g.wasm-component-ld
is in$PATH
). At this point there's feature parity withwasm32-wasi
plus a manualwit-component
run.std::net
build on WASI preview2wit-bindgen
wasi-libc
and use that in libstdwasm32-wasi-preview2
target into community crates as appropriate (e.g.libc
)-Zbuild-std
for a variety of applications/embeddings.std::net
implemented in the standard library.std::fs
andstd::env
as they exist today.getrandom
andlibc
, perhaps evenmio
as a stretch goal.wasm-component-ld
and its arguments and inputs finalized.wasm-component-ld
alongsiderust-lld
with the Rust compiler distribution.At this point, which is expected to be ideally no more than a year from Tier 3 acceptance, the intent is to apply for Tier 2 status for distribution through rustup to get more feedback for the component model an WASI.
Mentors or Reviewers
I plan on personally being on-the-hook for any reviews to the capacity that I'm qualified to perform such a review. I won't be able to review integration with the rest of the standard library since that falls under the libs implementation team, but I can review any WASI-specific parts, however.
Process
The main points of the Major Change Process are as follows:
@rustbot second
.-C flag
, then full team check-off is required.@rfcbot fcp merge
on either the MCP or the PR.You can read more about Major Change Proposals on forge.
Comments
This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.