search

rust-lang / compiler-team

A home for compiler team planning documents, meeting minutes, and other such things.
https://rust-lang.github.io/compiler-team/
Apache License 2.0
387 stars 69 forks source link

Promote wasm32-wasip2 to a Tier 2 target #760

Closed alexcrichton closed 4 months ago

alexcrichton commented 5 months ago

Proposal

This is a proposal to upgrade the wasm32-wasip2 target, (introduced here, and implemented here), to tier 2. Currently this is a tier 3 target that requires -Zbuild-std to work with.

This target is not easy to work with as a tier 3 target for a number of reasons:

It is possible to set everything up such that -Zbuild-std works but the lack of tier 2 support additionally makes it significantly harder to adopt this target throughout the ecosystem. For example when adding wasm32-wasip2 support to crates it's difficult to set up CI testing without a tier 2 target in place.

Recap of wasm32-wasip2 and other WebAssembly targets

The wasm32-wasip2 target is different from other WebAssembly targets in that the default output of the compiler is a component, not a core WebAssembly module. Support for components is not widespread at this time and is primarily found in Wasmtime for out-of-browser use cases of WebAssembly or jco for running in the browser.

The wasm32-wasip2 target reflects the development of WASI itself and is what the WebAssembly WASI Subgroup is working on these days. The wasm32-wasip2 Rust target will provide access via the standard library to appropriate WASI interfaces. The best example of this is that the wasm32-wasip1 target had only extremely basic support for networking (TCP/UDP) and it was not widely implemented in runtimes. With WASIp2, however, there are official interfaces for TCP/UDP/DNS/etc which are supported in WASIp2 implementations. This means that std::net works with the wasm32-wasip2 target. (modulo a final PR to get it hooked up to the support in wasi-libc).

For general intro-style documentation about components I'd recommend browsing https://component-model.bytecodealliance.org/

A new wasm-component-ld linker binary for wasm32-wasip2

This proposal intends to be a little more involved than "just" adding a new precompiled version of the standard library. This proposal is also to add the wasm-component-ld binary to the sysroot. This will live within the gcc-ld folder alongside the preexisting wasm-ld binary. The source of wasm-component-ld is intended to be modest where it's a wrapper around two pieces of functionality: the wasm-ld linker itself and the wit_component::ComponentEncoder type. While this could be integrated directly into rustc it's intended to be a separate binary to have a similar toolchain experience with Rust and C (e.g. this isn't getting integrated into Clang so it's a separate binary in the wasi-sdk).

This linker is currently shipping in the latest release of wasi-sdk and is how the Clang target for wasm32-wasip2 links code.

My plan at this time for the integration of this Rust-based binary is to publish wasm-component-ld to crates.io with a library crate that contains fn main() { ... }. This would be pulled in as a dependency as part of src/tools/wasm-component-ld in the Rust repository and then the build system would build it from there. That will enable vendoring/offline builds/etc to continue to work along with a defined location of where the dependency version is specified.

Tier 2 Requirements

From the tier requirements (with some omitted for brevity):

A tier 2 target must have value to people other than its maintainers

The wasm32-wasip2 target and the component model in general in WebAssembly have broad interest from many stakeholders. Much of this is embodied by the Bytecode Alliance and its member organizations.

A tier 2 target must have a designated team of developers

Myself and @rylev are signed up as target maintainers. If more interest is required I can ask around if other folks would like to sign up here too.

The target must not place undue burden on Rust developers not specifically concerned with that target.

This target will not have any more significant burden than the preexisting wasm32-wasip1 target. The wasm32-wasip1 target is already tested in CI to help weed out WebAssembly-specific issues and this target will mostly be new build configuration in the same builder handling the other wasm targets.

The target must provide documentation for the Rust community explaining how to build for the target using cross-compilation, and explaining how to run tests for the target

I will audit https://doc.rust-lang.org/nightly/rustc/platform-support/wasm32-wasip2.html if this proposal is accepted and implemented. Additionally the numerous tutorials for using Rust with the component model will eventually be updated to use wasm32-wasip2. For example cargo component will be updated to use this target when it's available.

The target must document its baseline expectations for the features or versions of CPUs, operating systems, libraries, runtime environments, and similar.

I'll audit https://doc.rust-lang.org/nightly/rustc/platform-support/wasm32-wasip2.html for this as well. This will follow the default set of WebAssembly features as other WebAssembly targets and the OS will be defined as the WASIp2 set of interfaces. The set of interfaces in WASIp2, and their iteration over time, is governed by the WebAssembly WASI Subgroup.

Tier 2 targets must not leave any significant portions of core or the standard library unimplemented or stubbed out, unless they cannot possibly be supported on the target.

This target will primarily omit the std::thread module when it is introduced. Not supporting std::thread matches wasm32-wasip1 and is generally not possible today. Work is happening within the component model and WebAssembly itself to eventually support threads.

If the target supports C code, and the target has an interoperable calling convention for C code

The wasm32-wasip2 is intended to interoperate with C code and this is tested in CI for wasm32-wasip1 and will be exercised in community crates.

The approving teams may additionally require that a subset of tests pass in CI, such as enough to build a functional "hello world" program

I am not planning at this time to add tests to CI for wasm32-wasip2. Instead I plan on only implementing build support and getting it into the ecosystem. Eventually in the long-term I'd like to migrate the testing of wasm32-wasip1 to wasm32-wasip2 as it contains the ability to run networking tests (std::net). This will take time to implement, however.

Building the target in CI must not take substantially longer than the current slowest target in CI, and should not substantially raise the maintenance burden of the CI infrastructure

A modest amount of time will be taken compiling wasm-component-ld for all host targets. This is expected to be negligible, however. Otherwise I plan on reusing the dist-various-2 builder for the target itself which as of the latest merge to Rust takes 55m compared to the full run time of 2h14m.

Tier 2 targets must not impose burden on the authors of pull requests, or other developers in the community, to ensure that tests pass for the target

I recently migrated testing of WebAssembly on CI from wasm32-unknown-unknown to wasm32-wasip1 with Wasmtime which provides a better experience with at least being able to print to stdout/stderr. I've not otherwise been alterted to difficulties in testing for WebAssembly. At this time wasm32-wasip2 is not expected to move the needle here much relative to the current state.

The target maintainers should regularly run the testsuite for the target, and should fix any test failures in a reasonably timely fashion.

Given the similarities of the wasm32-wasip1 and wasm32-wasip2 targets I wasn't planning for this at this time. My hope would be that the testing of wasm32-wasip1 is sufficient until the point at which wasm32-wasip2 might be run instead.

Mentors or Reviewers

I plan on personally doing the work to ship wasm-component-ld and enabling this as a tier 2 target.

Process

The main points of the Major Change Process are as follows:

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

rustbot commented 5 months ago

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections to the proposal should be discussed on Zulip and formally registered here by adding a comment with the following syntax: 

 @rustbot concern reason-for-concern 
 <description of the concern>

Concerns can be lifted with: 

 @rustbot resolve reason-for-concern

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler @rust-lang/compiler-contributors

sunfishcode commented 5 months ago

Edit: Moved to the zulip thread.

syrusakbary commented 5 months ago

A tier 2 target must have value to people other than its maintainers

The response to this question has been broadly generic "The wasm32-wasip2 target and the component model in general in WebAssembly have broad interest from many stakeholders", and I think is not properly aligned with the reality. Mainly, because the members of the Bytecode Alliance are actually the maintainers of WASI Preview 2.

I am personally not aware of a single user of Preview 2 outside of the maintainers of WASI Preview 2 (Fastly, Fermyon, Intel, ...). Not a single standalone runtime (outside of the BA), has adopted Preview 2 (that is: not wazero, not wamr, not stitch, not wizard, ...).

It has also been widely reported (1, 2) that WASI Preview 2 might not aligned with community interests, and maybe that's why is seeing so little adoption outside of the Bytecode Alliance.

Of course, my views can be limited and I might not be seeing the whole picture. Could you point of a maintainer outside of the Bytecode Alliance that maintains WASI Preview 2?

I hope my scepticism is interpreted with the best light. Because I deeply care about the Rust ecosystem, I do want to make sure if we lower the bar to accept targets to be Tier 2, this is done in a transparent fashion, in a way that tradeoffs on the requirements are analyzed and weighted properly.

Thanks!

alexcrichton commented 5 months ago

@syrusakbary issue comments here are reserved for procedural things, for other comments like yours you should use the Zulip stream. I've responded on Zulip

syrusakbary commented 5 months ago

Thanks @alexcrichton.

As I don't know the data retention policy of that Zulip channel and the post linked is already public, I'm reposting here the response in case is useful for readers in the future:

(alexcrichton) [...] I understood the term "maintainers" there to be the maintainers of the Rust target itself

(simulacrum) I'm pretty sure we've intended that to mean it's not just for the humans requesting addition but a broader group, and notably not a closed group. I don't think it's a very high bar to clear, basically it can't be a target that you e.g. can only develop on with a privately available toolchain or privately available hardware. (simulacrum) (and we semi-routinely drop targets if they become disused, so I think if p2 isn't actually stable etc. it's not a big deal from rust's perspective)

I don't currently have more questions on this matter, but if I do I'll ask them on Zulip first

davidtwco commented 5 months ago

@rustbot second

apiraino commented 4 months ago

@rustbot label -final-comment-period +major-change-accepted