Proposal: "Protobuf main"

[SoniEx2]

Instead of having the classic argc/argv interface which everyone knows and loves, it'd be nice if wasi could provide a typed CLI interface, either using protobuf or some bespoke new thing.

Specifically, something supporting tagged enums/sum types for arguments.

It should be as easy as specifying whether an argument is a file or a string or an open fd.

As a hypothetical example, a "protobuf main" grep might take an argument for a pattern. If this argument is a string, the pattern is the string. If this argument is a file or an open fd, the pattern is read from the file/fd.

Further type checking could be applied to such commands, which are just not available with current tools.

Something like

int main(wasi_protobuf_t protobuf)

would define a "protobuf main".

A few unresolved questions are:

1. How to communicate the schema to the host.
2. How to represent C argc/argv in protobuf space. (i.e. what restrictions do we apply to the schema? what elements does the host require the schema to support? can we just say "every schema that wants to support C/POSIX must have at least [this element]"?)

[SoniEx2]

(could this just use wit or something?)

(#pragma wit filename.wit?)

[pchickey]

This is more of a Component Model repo question, but the thinking right now is that, if you have some sort of foreign interface specified in an IDL, whether that be protobuf or any of the many other languages/tools in that space, you really want some sort of automated tool which translate that interface definition to a wit interface definition, and also generates a component which exports that interface, and imports wasi-sockets or wasi-http.

The interface translation and implementation would both have a lot of design latitude, so this isn't something we're thinking about implementing as part of the component model or wasi standards, but rather as a tool that is based on top of those standards which component authors wouldn't even necessarily need to know exists if they are just using the generated wit to create bindings, and using wasm-compose to link to the implementation.

[SoniEx2]

we don't know the appropriate repo.

we thought wasi was the part of wasm concerned with supporting C compilation. and since we want to see a C main based on a schema (as opposed to argv of strings), then it's relevant to whatever defines "C on wasm".

[pchickey]

Wasi is not a part of wasm concerned with supporting C compilation. It is a modular collection of interfaces for interacting with external systems, e.g. filesystems, http, and sockets. The readme has more information: https://github.com/webassembly/wasi#wasi-high-level-goals

C on wasm is not defined by any standard, but instead by whatever C toolchain you choose. There are many high quality C toolchains which target totally different wasm platforms, for example emscripten is used to target web embeddings, and wasi-sdk is used to target wasi embeddings. Both of those use clang under the hood, but with a different libc and set of assumptions about the embeddings.

To use the component model from C, you can use wit-bindgen to generate a C header, and compile that with wasi-sdk and run it in a WASI preview 2 engine: https://github.com/bytecodealliance/wit-bindgen#guest-cc. At this time, Wasmtime and jco are the only two engines which support preview 2.

[SoniEx2]

How would we use the component model for a wasm "binary" (with a C main function)? This isn't defined anywhere as far as we can tell. Finally, wasi would be responsible for defining binary calling (aka system(), posix_spawn(), etc), which we think makes it the relevant place.

[pchickey]

The component model defines how component types, including the types of import and export functions, work at the binary level, as well as how they are represented in wit files: https://github.com/WebAssembly/component-model/tree/main/design/mvp.

WASI's role is to specifies a number of component model worlds, which are designed to work on multiple providers/embeddings. One of those worlds is wasi:cli/command, specified in https://github.com/WebAssembly/wasi-cli/, which describes a set of imports functions available, and an export function run required from a component binary in order to run it as a "cli command". This is the world we expect many "legacy" applications targeting the subset of POSIX supported by WASI will use. Today, you can target this world by using wasi-sdk, along with wasm-tools's wit-component and wasmtime's wasi_snapshot_preview1.command.wasm adapter. This toolchain isn't yet polished and super accessible to users, but it does work in some tests in the wasmtime repo. wasi-libc, along with the wit-component and adapter chicanery, will hook up the user's int main(argv...) to be called underneath the run export.

Another world is wasi:http/proxy, specified in https://github.com/WebAssembly/wasi-http/blob/main/wit/proxy.wit, which provides a smaller set of imports, and expects an incoming-handler export https://github.com/WebAssembly/wasi-http/blob/main/wit/incoming-handler.wit, which is a function that takes an http response and a means to respond as an argument. This component is expected to be used in various serverless contexts, such as Fastly's Compute@Edge which I work on, along with many other providers. (None of this has shipped in production yet, but I think right now Fermyon has a prototype implementation that can actually execute the wasi-http proxy world, and we are working on landing that into wasmtime.) If you follow the same wasi-sdk, wit-component and adapter (except, use the wasi_snapshot_preview1.reactor.wasm instead) process, you'll also need to use wit-bindgen to generate a C header which provides all of the declarations corresponding to the wasi:http/proxy world (as well as a .o containing some component types magic), and then define the incoming_handler function declared in that header.

Component composition means that you don't need to limit your programs to the interfaces described by the WASI standards. You can (and should!) define your own interfaces using wit. wit-bindgen allows you to translate that wit into imports and exports declarations which you can use in your own source code. Then, you can either add a host implementation of that interface to a wasm component engine, e.g. use wasmtime::component::bindgen! and define your interface using Rust code that will be used directly by wasmtime. Or, alternatively, you can implement the interface entirely with another wasm component in terms of some common host-provided interface like wasi-http, and use wasm-compose to link it to the component which needs that interface.

[SoniEx2]

okay but that's missing the point.

how dynamic is wit? how introspectable is wit? can you use wit at the OS command interface (IPC) level?

given a problem like the one described here: https://fedi-to.net/go?h=5&target=web%2Bap://hachyderm.io/@NireBryce/110300287137899566 what can wit do about it?