all: add GOOS=wasip1 GOARCH=wasm port

[johanbrandhorst]

Background

The WebAssembly System Interface (WASI, https://wasi.dev/) is gaining popularity as a compile-once-run-anywhere target for developer and cloud native applications. Many cloud providers are offering services that make it possible to execute WASI directly inside familiar orchestration frameworks like Kubernetes (https://learn.microsoft.com/en-us/azure/aks/use-wasi-node-pools, https://docs.krustlet.dev/howto/), or on edge compute platforms (https://developer.fastly.com/learning/compute/, https://blog.cloudflare.com/announcing-wasi-on-workers/) and the popular developer tool Docker has beta support for executing wasi directly (https://docs.docker.com/desktop/wasm/). For Go to remain relevant in a hypothetical world where this becomes a significant part of software delivery, it must support compiling code to the Wasm binary format and the WASI syscall API.

Proposal

We propose adding a new port, GOOS=wasip1 GOARCH=wasm, that targets the wasi_snapshot_preview1 syscall API. We further propose allowing the use of the go:wasmimport compiler directive in the syscall package, in addition to the currently allowed runtime and syscall/js packages.

Discussion

Go already supports WebAssembly (Wasm) through the existing GOOS=js GOARCH=wasm port, and the implementation of this proposal would reuse the existing Wasm architecture code and change the interface with which the compiled code interacts with the outside world. It builds on the accepted proposal (https://github.com/golang/go/issues/38248) for a go:wasmimport compiler directive for defining Wasm host function imports. The compiled code would be a “Command”, executing func main and running until exit, similar to the existing js/wasm port.

Syscall API target

Today, implementing WASI means implementing the wasi_snapshot_preview1 API described in the spec. However, this interface is evolving without the insurance of backward compatibility. A “preview2” version is already being worked on. Should the Go compiler support the old one for now, and switch to the new one in the future, or should we name the new GOOS such that we can add new GOOS’s for new WASI APIs? We propose that we assume the wasi_snapshot_preview1 API for now and that future releases of Go may add support for newer syscall APIs under a different GOOS (e.g. GOOS=wasip2 for wasi_snapshot_preview2).

Maintainers

Since this is a new port and the porting policy requires at least two maintainers, Evan Phoenix (@evanphx), Julien Fabre (@Pryz) and I (@johanbrandhorst) are volunteering to be maintainers of this port.

Testing

The wasi/wasm port will be tested by executing the standard library tests using an established WASI VM, such as Wasmtime. This software has precompiled binaries available for download, which can be used to set up a builder for the trybots, similar to how NodeJS is used for the js/wasm port.

What happens to the js/wasm port?

The existing js/wasm port will remain relevant for the purposes of compiling Go Wasm for running in a JavaScript VM and using the syscall/js interface for interacting with the JS world. Both ports will coexist, and should eventually require minimal differences in compiler and syscall code. See the discussion on rewriting wasm_exec.js for more information.

A note on capabilities

wasi_snapshot_preview1 is limited in ways that may be surprising to users, for example, it is not possible to open a network socket with the APIs defined in the spec. The initial implementation of the wasi target will aim to implement as much of the standard library as possible, but there will be big gaps.

Related issues

This would close #31105, which has mostly been a discussion issue.

Future work

WASI Preview2

As the second snapshot of the WASI standard matures, we will aim to add support for the new standard. This will unlock new functionality such as networking sockets and ensure that Go’s WASI support remains relevant for users. This could be done in any new major release of the Go toolchain, but not in a minor revision.

Considering the upcoming changes in preview 2, it is a legitimate question to ask whether the work to add support for wasi_snapshot_preview1 is worth doing; could we simply wait for the next standard iteration? We believe the work to be useful because at this time, all compilers are targeting preview 1, and preview 2 seems far from being fully completed. We also believe that runtimes will provide polyfills to preview 1 while preview 2 is in the process of being implemented (see this Wasmtime issue). Finally, the next version of the WASI standard is split into multiple components, and it is possible that specifications for each component will be finalized at different times, with preview 1 remaining the de-facto fallback for components that are not yet fully specified or implemented by runtimes yet.

Rewriting wasm_exec.js as WASI and unifying syscall interfaces

The existing js/wasm port has a custom syscall interface implemented by wasm_exec.js and run on any JavaScript VM. Now that a standard is emerging, the js/wasm target should reuse the same syscall interface, allowing parts of the syscall interface between wasi and js to be unified to reduce maintenance burden. This would require implementing a WASI interface shim in wasm_exec.js, which is a significant undertaking, and thus out of scope of this initial WASI work.

WASI Libraries (AKA Reactors)

The WASI concept of libraries allow compiled binaries to expose single functions for consumption from the host. This is not something that will be supported in the initial WASI port, as it requires a concept of marking Go functions as exported (i.e. //go:wasmexport), and somehow facilitating the execution of a single function. For more discussions on why this is complicated, see #42372.

GOOS=none GOARCH=wasm

Binary wasm can run without any particular knowledge of its host, perhaps using something like GOOS=none, similar to Rust’s wasm32-unknown-unknown target. This proposal does not propose any such port be added, but it may be something to consider in the future. The name “none” is, of course, not decided.

Authors

@johanbrandhorst, @Pryz, @evanphx, @achille-roussel

[prattmic]

I like the idea of using GOOS=wasi to distinguish from GOOS=js. It seems like a natural place to select wasi.

The obvious concern here to me is the instability of the API of wasi_snapshot_preview1. Is wasi_snapshot_preview2 planning to be backwards compatible with wasi_snapshot_preview1 binaries? (It sounds like no?). If not, I am concerned by GOOS=wasi changing APIs between versions because I suspect (a) some users will want preview2 ASAP to use new feature, and (b) some users will want to keep preview1 because their wasm runtime doesn't support preview2 yet. These are in direct conflict with one another.

As a workaround, we could have GOWASI=preview1, or something like that, similar to GOARM, GOAMD64, etc.

What is the timeline for a "stable" version of WASI? The other end of the spectrum would be to say that GOOS=wasi is not stable (hidden behind a GOEXPERIMENT maybe?) and will change compatibility arbitrarily from release-to-release until there is a stable version of WASI to target.

[prattmic]

From https://github.com/WebAssembly/WASI: "The WebAssembly System Interface is not a monolithic standard system interface, but is instead a modular collection of standardized APIs. None of the APIs are required to be implemented to have a compliant runtime. Instead, host environments can choose which APIs make sense for their use cases."

Will there be some minimum requirements that the Go runtime will require from the host environment? Or will Go still work even if the host environment provides no APIs?

[codefromthecrypt]

@prattmic while not documented really, the usual way features are disabled is via syscall.ENOSYS errors.

For example, when source is compiled with GOARCH=wasm GOOS=js and let's say wazero is running it, if that code tries to access the filesystem and the filesystem is disabled, the wazero host functions return an ENOSYS error which the compiled code expects and returns an error message like "not implemented in js"

In the case of WASI, and specifically the most implemented version of it (snapshot-01), there are error codes which map to syscall.Errno here https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md#-errno-enumu16

[johanbrandhorst]

The obvious concern here to me is the instability of the API of wasi_snapshot_preview1. Is wasi_snapshot_preview2 planning to be backwards compatible with wasi_snapshot_preview1 binaries? (It sounds like no?). If not, I am concerned by GOOS=wasi changing APIs between versions because I suspect (a) some users will want preview2 ASAP to use new feature, and (b) some users will want to keep preview1 because their wasm runtime doesn't support preview2 yet. These are in direct conflict with one another.

The existing js/wasm port has generally taken a conservative approach to including new features, and we would seek to emulate that. We don't yet know what would be the threshold for switching over to preview2, but it would likely be year(s) in the future. A GOWASI environment variable as suggested may be considered, but it shouldn't be necessary in the near term.

I will note also that the existing js/wasm port is still considered experimental and can introduce breaking changes at any time. The wasi/wasm port would similarly not provide any backwards compatibility guarantees. It would seem appropriate for this to remain the case until a stable WASI API spec is available and implemented in runtimes at least.

Will there be some minimum requirements that the Go runtime will require from the host environment? Or will Go still work even if the host environment provides no APIs?

Go binaries compiled with GOOS=wasi would require the host to provide the full wasi_snapshot_preview1 API. A hypothetical GOOS=none GOARCH=wasm could be introduced to avoid any host dependencies, but it's not currently part of our planning.

[prattmic]

@codefromthecrypt:

@prattmic while not documented really, the usual way features are disabled is via syscall.ENOSYS errors.

@johanbrandhorst:

Go binaries compiled with GOOS=wasi would require the host to provide the full wasi_snapshot_preview1 API.

These seem a bit contradictory?

I'm specifically wondering about APIs which the Go runtime cannot run without at all. e.g., we may require clock_get_time to implement runtime.nanotime, because the runtime more-or-less can't run without a source of time [1]. I'm wondering if there are other cases.

It doesn't seem like we'd need to require all APIs. e.g., fd_sync wouldn't be called by Go unless the program explicitly calls os.File.Sync/syscall.Sync. Those returning an error seems fine to me.

[1] OK, maybe not a perfect example, since technically -tags=faketime doesn't require a time source.

[johanbrandhorst]

I'm not sure what you're asking exactly, the way I see this being implemented is by translating syscalls in the code to the relevant host API calls. Sure you could build a wasi binary that doesn't use all of the API and it'd work fine on a host that only implements the part of the API that's used, but I don't think the implementation should need to do any sort of feature capability negotiation with the host - if the API returns ENOSYS then the function call fails up the stack. Does that sound okay?

For your specific example, I guess if clock_get_time isn't implemented it would fail very quickly at runtime, not try to get by without a clock source.

[ianlancetaylor]

At the very least for documentation purposes I think we want to be able to write down which APIs must be implemented in order to run simple Go programs.

[johanbrandhorst]

At the very least for documentation purposes I think we want to be able to write down which APIs must be implemented in order to run simple Go programs.

Not that I disagree, but I'm a little confused by this inquiry - are we expecting users to implement their own partial implementations of wasi_snapshot_preview1 in the hopes of running simple Go programs? It's tempting to say that any Go compiled wasi binary requires the host to provide the API defined by the spec. What is the purpose of defining a minimal API used by simple Go programs?

[codefromthecrypt]

@ianlancetaylor

At the very least for documentation purposes I think we want to be able to write down which APIs must be implemented in order to run simple Go programs.

I think initially it would look like TinyGo, which implements a subset of wasi. Here's a list of functions that are used and who uses them https://wazero.io/specs/#wasi and here's an example simple cat program. Hope it helps!

$ wasm2wat ./cmd/wazero/testdata/cat/cat-tinygo.wasm|grep 'import "wasi'
  (import "wasi_snapshot_preview1" "fd_write" (func $runtime.fd_write (type 0)))
  (import "wasi_snapshot_preview1" "clock_time_get" (func $runtime.clock_time_get (type 1)))
  (import "wasi_snapshot_preview1" "args_sizes_get" (func $runtime.args_sizes_get (type 2)))
  (import "wasi_snapshot_preview1" "args_get" (func $runtime.args_get (type 2)))
  (import "wasi_snapshot_preview1" "proc_exit" (func $runtime.proc_exit (type 3)))
  (import "wasi_snapshot_preview1" "environ_get" (func $__imported_wasi_snapshot_preview1_environ_get (type 2)))
  (import "wasi_snapshot_preview1" "environ_sizes_get" (func $__imported_wasi_snapshot_preview1_environ_sizes_get (type 2)))
  (import "wasi_snapshot_preview1" "fd_close" (func $__imported_wasi_snapshot_preview1_fd_close (type 4)))
  (import "wasi_snapshot_preview1" "fd_fdstat_get" (func $__imported_wasi_snapshot_preview1_fd_fdstat_get (type 2)))
  (import "wasi_snapshot_preview1" "fd_filestat_get" (func $__imported_wasi_snapshot_preview1_fd_filestat_get (type 2)))
  (import "wasi_snapshot_preview1" "fd_prestat_get" (func $__imported_wasi_snapshot_preview1_fd_prestat_get (type 2)))
  (import "wasi_snapshot_preview1" "fd_prestat_dir_name" (func $__imported_wasi_snapshot_preview1_fd_prestat_dir_name (type 5)))
  (import "wasi_snapshot_preview1" "fd_read" (func $__imported_wasi_snapshot_preview1_fd_read (type 0)))
  (import "wasi_snapshot_preview1" "fd_seek" (func $__imported_wasi_snapshot_preview1_fd_seek (type 6)))
  (import "wasi_snapshot_preview1" "path_open" (func $__imported_wasi_snapshot_preview1_path_open (type 7)))

[dmitshur]

CC @golang/release.

[prattmic]

At the very least for documentation purposes I think we want to be able to write down which APIs must be implemented in order to run simple Go programs.

Not that I disagree, but I'm a little confused by this inquiry - are we expecting users to implement their own partial implementations of wasi_snapshot_preview1 in the hopes of running simple Go programs?

I haven't been following wasm/wasi super closely, so maybe I've misunderstood, but I thought that wasm/wasi was commonly using in "sandboxing"-type scenarios, where presumably the operator wants to limit the APIs that the sandboxed program has access to. Documenting the minimum requirements for the Go runtime itself makes it clear to those building a sandbox what the most restricted set of APIs they can provide is (and if that is still too permissive, maybe Go programs aren't a good sandboxee target).

[johanbrandhorst]

I haven't been following wasm/wasi super closely, so maybe I've misunderstood, but I thought that wasm/wasi was commonly using in "sandboxing"-type scenarios, where presumably the operator wants to limit the APIs that the sandboxed program has access to. Documenting the minimum requirements for the Go runtime itself makes it clear to those building a sandbox what the most restricted set of APIs they can provide is (and if that is still too permissive, maybe Go programs aren't a good sandboxee target).

This is a great point, and I agree. Thank you. Do we have a rough idea of the syscalls required by the runtime today? I expect to get exact information would require an experimental implementation (which we are working on), but I could add a preliminary list to the proposal. The TinyGo information is presumably not going to be representative of the behavior of gc?

[prattmic]

I think it is fine to figure out in prototyping, the list doesn't need to be in the proposal.

FWIW, quickly looking through https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md#modules, the only ones that look really important to me are clock_get_time, and proc_exit. The args and environ ones could be optional, but the implementation will need to be careful to check for errors (it would be easy to assume they wouldn't fail).

Some sort of I/O (fd_read, fd_write) shouldn't technically be required, though in practice almost any program probably wants to use stdin/stdout/stderr.

[prattmic]

Hm, one more problem I see is that https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md#modules does not seem to implement any kind of timer/sleep/wait. Am I missing something? It seems like that will require the Go runtime to spin when there is nothing else to do.

[evanphx]

@prattmic There is poll_oneoff which includes both clock and fd event types, so we'd use that. https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md#-eventtype-enumu8

[johanbrandhorst]

The latest CNCF annual survey describes WebAssembly as "the future", though it is unclear whether that's within a JS runtime environment or WASI.

[Mossaka]

Hey there, I am very excited for this proposal!

We also believe that runtimes will provide polyfills to preview 1 while preview 2 is in the process of being implemented

This is indeed true. Please see this repo where the community is building a polyfill adapter for wasi preview1 modules to call preview2 functions.

[rsc]

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

[johanbrandhorst]

I just made a minor change to the proposal. In addition to the previous statement, I added

We further propose allowing the use of the go:wasmimport compiler directive in the syscall package, in addition to the currently allowed runtime and syscall/js packages.

This will be necessary to define the syscall methods used to interact with the WASI host through the go:wasmimport compiler directive.

[aclements]

The compiler/runtime team at Google is generally supportive of WASI support. It seems like a good idea and an important direction for WASM.

Our one big question is how to set things up for preview2 in the future. @prattmic suggested a GOWASI environment variable to parallel GO386/GOAMD64/etc. That would be our first GO$GOOS variable, and it's not "additive" in the way that the GO$GOARCH variables are (e.g., amd64 but with more instructions), but it's certainly worth considering. Another option is that we put it in GOOS itself, like GOOS=wasip1. My impression is that WASI preview1 and WASI preview2 kind of are different operating systems, rather than a base layer with additions in preview2, which suggests using different GOOS values to me.

[rsc]

We already adapt OS implementations based on what's available, like using newer functions on Windows if they are present in the DLL, or falling back to older system calls on Linux when a newer one returns ENOSYS. Do we think preview1 and preview2 will be close enough to make that same approach feasible?

One possibility is to use GOOS=wasi for preview1 and then decide when preview2 is more settled whether that needs to be a separate GOOS or can be incorporated into GOOS=wasi.

[codefromthecrypt]

Do we think preview1 and preview2 will be close enough to make that same approach feasible?

TL;DR; I think they are too different because component model changes things. We should not assume the same approach will be best both for snapshot01 and snapshot02

WASI preview1 was a continuation of CloudABI and preview2 is a complete rewrite.

preview1 has a relatively straightforward, albeit monolithic ABI. All functions are in the same wasi_snapshot_preview1 and only depend on WebAssembly Core 1.0 features.

what's being called preview2 is a complete redo, which is based on the finalizing component model. Component model is a change to the binary format of WebAssembly. Specific to WASI, this splits various modules such as wasi-filesystem. There will be an adapter to forward snapshot01 to this new model.

For what its worth, the wazero team will be implementing component model and WASI snapshot 2 when they are ready, just likely not until the end of the year. As we learn more, we can help advise.

[evanphx]

as @codefromthecrypt said, preview1 and preview2 are like different OSs. Much like linux and solaris, there are some syscalls that happen to be similar, but the complete set of syscalls is quite different and the semantics around similars ones also differs.

There is going to be concept of "worlds" in preview2 it sounds like, which are collections of functionality various runtimes will implement. Some worlds will resemble what we currently see in terms of preview1, effectively a posix look-alike system. Some worlds are looking to have higher level functionality such as cryptography, http, etc.

Eventual support for preview2 will likely come in support for a fixed set of preview2 worlds, since these worlds are where the real specifications will be concrete.

While I'd love the ability to detect at runtime which set is being used, it doesn't look like that's being proposed currently. And doing so would mean, mostly, that every compiled Go binary would contain a full preview1 and preview2 impls, switched at runtime.

Given that, I'm leaning towards switching on GOOS, with preview1 coming in as wasi or wasi-p1. When preview2 shakes out, we'd probably name the GOOS string along the lines of the world that is supported, wasi-posix or something.

Lastly, I'm unsure of the history behind an idea like this, but we could have GOOS=wasi change overtime as preview1 fades and preview2 moves on, with GOOS=wasi targeting the "mainline" version, with an aim of having GOOS=wasi being the only target once WASI gets out of preview states.

[ianlancetaylor]

Yes, we can in principle change GOOS=wasi to start as preview1 and then to switch to preview2. Of course this would mean that people using the new Go release would no longer be able to use preview1. If we expect everybody to quickly move from preview1 to preview2 then that might be OK. Depends on how people use it.

[evanphx]

@ianlancetaylor Ah, I forgot an important bit! I was thinking that GOOS=wasi would select the "primary" implementation, but that we'd still have GOOS=wasi-preview1 or something to explicitly select which WASI.

[ianlancetaylor]

That is also possible, though it may be somewhat painful to support two different variants simultaneously.

[evanphx]

@ianlancetaylor Oh for sure, that's part of the question we're asking: how many different WASI variants should be support simultaneously.

[johanbrandhorst]

I think my preference is to have a single GOOS=wasi, the meaning of which changes over time. We'd adopt preview2 once a suitable number of runtimes were shown to support it, and drop preview1 support from gc. Similarly, once preview3 and later 1.0 become widely adopted, we'd drop support for preview2 and so on. We'd have to communicate this clearly so that users can make an informed choice about whether to use wasi with Go.

The expectation within the wasi community to me seems to be that runtimes will update to the latest preview reasonably quickly and provide "polyfills" so that they can execute binaries compiled for previous wasi versions. I expect our users to want us to err on the side of moving faster with wasi at the cost of long term compatibility.

Assuming runtimes will be upgrading to support both preview2 and preview1 soon after preview2 is stabilized, there should be little reason for Go to provide preview1 builds, particularly when those builds are so limited in capabilities (no sockets, for example).

[ghost]

WASM can be safely ignored. There are no cloud portability issues in Go.

Web performance issues should be solved in other way.

[x1unix]

Would also appreciate an ability to export Go functions as WASM export function.

[achille-roussel]

@x1unix I believe this is the issue you might want to follow regarding exporting Go functions in WASM modules https://github.com/golang/go/issues/42372

[leaanthony]

I think my preference is to have a single GOOS=wasi, the meaning of which changes over time. We'd adopt preview2 once a suitable number of runtimes were shown to support it, and drop preview1 support from gc.

Wouldn't that essentially be a breaking change? Or is the idea that until there's a non-preview wasi specification, this should be considered unstable?

[codefromthecrypt]

One way to put it is these two wasi operating systems will be even more different from each other than darwin and freebsd, so the "different os" approach suggested by @evanphx in https://github.com/golang/go/issues/58141#issuecomment-1423114723 seems appropriate.

Concrete suggestion:

GOOS=wasi-snapshot-01 named based on the tag which defines it. https://github.com/WebAssembly/wasi/tree/snapshot-01

This will forever be the widely implemented current WASI, and this will stick around well past the other. I don't expect runtimes to remove this for a very long time.

GOOS=wasi-components for the modular thing that is when discussed as a monolith called snapshot or preview 02 (and 03) right now. This would be comprised of many wasi modules including wasi-filesystem, wasi-crypto etc. We can keep an unstable GOOS value for this until it evolves.

This will help surprising breaks IMHO, and also clarify the wildly different tracks of work and code. The former is implementable today and will drop into existing work in docker etc. The latter will be able to evolve over time, named differently as they are different operating systems even if owned by the same W3C subgroup (wasi)

[x1unix]

@codefromthecrypt it's possible to use the same approach to spec versioning as it's implemented with ARM arch version and GOARM variable.

Smth. like GOWASI env variable

[johanbrandhorst]

I think my preference is to have a single GOOS=wasi, the meaning of which changes over time. We'd adopt preview2 once a suitable number of runtimes were shown to support it, and drop preview1 support from gc.

Wouldn't that essentially be a breaking change? Or is the idea that until there's a non-preview wasi specification, this should be considered unstable?

Yes, this would be a breaking change, and yes, the wasi/wasm port will remain experimental (and not bound by the Go 1.0 compatibility promise) until we have a stable wasi API, at least. This is already the case with the js/wasm port.

GOOS=wasi-snapshot-01 named based on the tag which defines it.

I can see the rationale behind this approach, but I humbly submit that it will be too much work maintaining two different APIs (and more!) going forward. Evolving the meaning of GOOS=wasi during this ecosystem-wide phase of instability seems appropriate to me. When we have a stable wasi API, and there is a proposal for a new, say, wasi v2, we could consider adding a GOWASI variable for specifying the specific target version. All these preview APIs are (hopefully) going to be a footnote in the long term use of wasi as a platform. If we commited to all these different GOOS's, we're going to make users depend on this functionality, which we will eventually want to phase out, ending up breaking users anyway. I will reiterate that our preview1 implementation will likely forgo networking as a whole, and users will be begging us to implement preview2 as soon as it is provided by enough runtimes.

[codefromthecrypt]

FWIW GOWASI sounds fine to me. I guess all we are talking about is where to stick the dimension, and this is better than hyphens, IMHO. I personally might recommend people set that always to what they need as the ABIs again have nothing to do with each other.

P.S. I don't think existing wasm ecosystem will be concerned about lack of sockets, as they are used to it. This indeed will be a concern for newcomers. Existing folks will rejoice they can use more standard libraries, like json, which currently don't work in tinygo. So in some ways, you'll have some relief that things work and stress to make more things work indeed :D

[Pat3ickI]

Does this apply for Cgo too ?

[evanphx]

@Patrickmitech as in, will go complied against WASI be able to use cgo to call other code compiled with webassembly? There are currently no plans for that.

[aclements]

There's a fairly high bar to adding a GOWASI. It would be our first GOOS versioning variable. And now that I understand how different preview1 and preview2 are, this would be quite unlike the GO$GOARCH variables. For example, we use GOARCH=arm64 for 64-bit ARM, rather than GOARCH=arm GOARM=8, even though ARM considers that to be "v8" of the ARM ISA. Because there's really nothing shared between arm and arm64, we treat them as different GOARCHes. The same applies to WASI.

I think the question remains whether we support multiple GOOS values for different WASI previews. It sounds like, given the ecosystem expectations, there's not much need to support multiple versions simultaneously.

[rsc]

It sounds like for preview2 either we should

• Change GOOS=wasi to mean preview2, assuming that preview1-only environments will disappear
• Or add GOOS=wasi2, to keep the two separate
• Or add a GOWASI=preview2, even though it might be a completely different "OS"

We don't know enough about preview2 to really guess what the right option is. No matter which of these we do, it sounds like people are okay with GOOS=wasi starting out as meaning preview1, and we can decide what the next step is when preview2's rollout becomes clearer.

That is, I don't think we have to decide what we do for preview2. We can say that GOOS=wasi is good enough for preview1 and worry about preview2 later.

Do I have that right?

[johanbrandhorst]

I think that sounds right, my only reservation being that we'll almost certainly want GOOS=wasi to mean the stable version of WASI when that comes around, so GOOS=wasi meaning preview1 should not be the case perpetually.

[codefromthecrypt]

I fear for lack of concrete details, folks are forced into "what seems to make sense". However, there are details which I believe lead directly and absolutely to a solution which doesn't believe there will be a simple cutover, especially not on preview2.

For example the below shows that the model used in preview2 is incomplete and will change once other things complete in preview3. This means preview2 and preview1 will absolutely overlap.

Streams

Streams are expected to be available in the Preview 3 timeframe, as part of the Component Model async proposal

As a temporary workaround for use cases that need byte streams, use the input-stream and output-stream types defined in wasi-io

Moreover, the models are extremely different in practice. For example, the below shows that in the case of preview1 a directory read affects the file descriptor of the directory, while preview2 a handle where there could be multiple possibly concurrent ones.

preview1 reading a directory (wit format):

;; fd_readdir(fd: fd, buf: Pointer<u8>, buf_len: size, cookie: dircookie) -> (errno, size)
;;
;; note: the size result is handled by the last parameter, which is a
;; memory offset to write the size (out-param).
(import "wasi_snapshot_preview1" "fd_readdir"
  (func (param i32 i32 i32 i64 i32) (result i32)))

current version of preview2 reading a directory (wit format):

;;
;;    read-directory: func(
;;        this: descriptor
;;    ) -> result<directory-entry-stream, error-code>
;;
;;    read-directory-entry: func(
;;        this: directory-entry-stream
;;    ) -> result<option<directory-entry>, error-code>
;;
;;    drop-directory-entry-stream: func(this: directory-entry-stream)
;;
;; note: I'm not really sure yet how the error code works as none of
;; the below have wasm results. Guessing the second param is an
;; out-param poked to see if it is a stream or an error.
(func (import "wasi-filesystem" "readdir")
  (param i32 i32))
(func (import "wasi-filesystem" "read-dir-entry")
  (param i32 i32))
(func (import "wasi-filesystem" "drop-dir-entry-stream")
  (param i32))

I'm not trying to be a stick in the mud, but I really do not believe preview2 is a LTS, maybe not even preview3. preview1 is, if for no other reason than by accident. In any case, I hope this info shares why I encourage us to consider this as a long term concern. Doing two wasi gives a lot more flexibility to roll with the eventuality of a drifty wasi2 while still allowing users of today a stable compilation target. Having a stable wasi of today is very much enabling and should continue to work for a long time. Otherwise, we may re-enter the position of go not being a suitable compiler, as I have no doubt other compilers will continue to support wasi 1 for a long time.

[johanbrandhorst]

I'm not sure why the degree of difference between preview1 and preview2 should matter to the user, to them ideally the only thing they should notice is that they can do more things with their Go code than they could when we supported preview1. Are you suggesting that we might cause subtly breaking changes when switching from preview1 to preview2? I could see that possibility.

I also don't see why preview2 not supporting streams implies that preview1 and preview2 should overlap. My understanding was that preview2 is essentially a superset of the functionality of preview1. Are you suggesting that there is functionality in preview1 that is not supported by preview2, such that a user may want to choose to compile from preview1 instead of preview2?

I appreciate your expertise in this area, I hope I don't come off as dismissive, but some clarification would be great 😁.

[codefromthecrypt]

No problem, and thanks for asking @johanbrandhorst FWIW I don't think you are being dismissive, rather doing the right thing and arguing for less code, or at least a chance of it. I want that, too, just I'm not sure it is realistic to put a near term date on that.

On current vs next features, there are changes are not only in the functions supported, but also what they support as parameters. Also, how concepts like how main works and pre-opens are handled (command world) are very different. To inventory this while the specs change even in format routinely is possible, but I would say from what I've seen the various sub specifications wasi next specifies more functions than wasi 1, except with more constrained parameters.

For example, you can see if a file is readable, but you can't get the posix 755 etc data back. It would really take a paper to go through all the differences, and I won't have time to do that for a month. That said I'm interested in it and will do it eventually for my own sanity.

I guess the part you are concerned about is if feature gaps between the two will be the primary user concern. As hinted above, things have to settle to even know what will be a feature, and I think if I can't figure it out, end users will have a harder time ;)

I think users will be driven by product support, as compiling to wasm only matters where it can be run. There are many existing wasi preview1 products out there, including envoy, various k8s tools, etc. An extremely relevant reason for users to care is to continue using a product which may have no roadmap or plan to support wasi-2 for the next couple years.

This may be due to an explicit support decision or a runtime constraint, or due to not wanting to have multiple cores. For example, some language compilers may choose to never do wasi-2 due to its complexity and incompatibility with the only w3c REC of the core: 1.0. Let's remember that component model literally changes the format of the webassembly binary. Also keep in mind that even component model isn't scheduled for the 2.0 draft of wasm yet!

Anyway, let's say the specs settle and runtimes immediately support it and products and other compilers immediately do also, and it is 2025 all products run both current wasm and component model based wasm, and no one needs to recompile current wasm anymore. As a software maintainer, I would be all for removing that code! I don't know if the year is 2025, but would bet at least $5 it isn't gonna be 2024 ;)

[codefromthecrypt]

I have an idea. the main change in wasi next is that it eventually uses component model, which is a different binary format than wasm 1.0 which is what go supports right now. What if instead of push and pull about the GOOS "wasi" we make this a decision about GOARCH where if the GOARCH is a choice that supports component model (writing 1.0 incompatible wasm), a GOOS "wasi" would be the other codebase. In other words, don't allow wasi 1 if compiling to component model. Does this help?

[codefromthecrypt]

concrete proposal

GOARCH=wasm GOOS=wasi == Wasm core 1.0 with wasi snapshot01 GOARCH=wasm2 GOOS=wasi == Wasm core 2.0 + component model with wasi 2,3,..1.0

[johanbrandhorst]

After speaking with @codefromthecrypt and others it seems clear to me that there is some anxiety around the planned adoption schedule of preview2. I wanted to add to the discussion that I think it's entirely possible (and perhaps likely) that we wouldn't implement preview2 or even preview3 and wait for 1.0, depending on the state of the ecosystem. It seems there is some question around the big changes from preview1 to preview2 which may cause regressions and instability.

Any discussion to move from one syscall interface to the next (i.e. preview1 to preview2) would be carefully considered with community feedback taken into account. We may even produce prototype implementations of these so that we can gauge the impact of switching to the new syscall interface.

Given that, is there still opposition to using GOOS=wasi to mean preview1 now and something else in the future (perhaps not until wasi 1.0), without the need for alternative GOARCH or GOWASI values?

[rsc]

GOOS=wasi would be targeting specific wasi runners (not browsers). Presumably those will have their own compatibility windows where they support, for example, both preview1 and preview2. It seems OK to change the specific target from one Go release to the next as long as we are clear in documentation that this will happen and perhaps also pre-announce the change. It will just mean people need an up-to-date-enough runner to update to a newer Go.

If we decide that GOOS=wasi is going to be a moving target at least through 1.0, have all concerns with this proposal been addressed?

[codefromthecrypt]

If we decide that GOOS=wasi is going to be a moving target at least through 1.0, have all concerns with this proposal been addressed?

@rsc the main concern of a moving target is that it will create some difficulty for people to transition. there will be a large gap (years) between the beginning of preview2 and the end of 1.0, and meanwhile a lot of platforms aren't likely to have the host functions for the interim steps. Also, there will likely be bugs both in compilers and runtimes as people figure out what preview 2..1.0 means in practice.

Given Go's 6 month release schedule the impact I'm most concerned with is people being pinned to an old Go, which will drop out of support before wasi 1.0 completes. In other words at the point a hard swap of GOOS=wasi meaning something less supported or stable than now, users will get pinned to a go compiler that still works with current day wasi.

Having a strict policy of only one operating system usable outside of browsers at a time will cause some problems whenever implemented, as not everything updates, in other words. The other problem is delayed feedback into the next version of wasi. For example, there's a large chance some design concern there goes unnoticed until too late. The chicken-egg problem forced when we have a policy of a single wasm operating system for outside browser use.

If we must do this, and I understand the valid reasons why... I believe the next wasi shouldn't cut over until 1.0 is reliably within year of completion. That won't be this year.

[rsc]

It sounds like maybe we do need to distinguish between the different wasi targets, at least until they get to 1.0. We don't want to have a GO$GOOS, so for now it seems like maybe separate GOOS are the answer. That would mean the initial version is GOOS=wasip1, then preview2 is GOOS=wasip2, and maybe finally 1.0 can be GOOS=wasi.

Do I have that right?