Open alexcrichton opened 1 year ago
Really nice job writing up this proposal! This all sounds pretty great to me.
One small question I had where I might just be missing the whole picture is: if it's optional to use use
to refer to other interfaces in the same package, then it seems like it'll lead to irregular use and so maybe we should either make it mandatory or disallowed.
Secondly: a small nit just on the Wat syntax at the end: to disambiguate interface-imports from future implementation-imports (which will include versions but also other optional sub-fields like version ranges, content-hashes and (real) URLs), maybe the Wat syntax is:
(import (interface "wasi:http/types" (version "0.2")) (...))
so that we can unambiguously add (import (implementation ...) (...))
later and also easily distinguish indicate where to use the grammar of interface IDs vs just kebab-names. Does that make sense?
This looks like a great step in the right direction to me. Feedback roughly in order:
The first change I'd like to propose to WIT is a requirement that all *.wit files will start at the top with a package ... statement.
If we have packages with multiple .wit
files, is the expectation that every single file would have a package
signifier at the top? My main concern with that would be redundancy. Bumping the version might mean changing the version in many different files.
I'd be interested to hear what you had in mind. Could we perhaps have just one WIT file that gets to represent the entire package or something similar? Perhaps a validation rule can just ensure only one of the WIT files has this package signifier?
Packages can consist of multiple *.wit files but the interfaces/worlds in a package are unioned together. This means you can't define interface foo in two files in the same package. Packages can always be represented with one file.
This seems like a nice simplification for now. Is the assumption then that duplicate names would just throw? Are all WIT files effectively globbed in the package directory and then deduped upfront, or are they only pulled in when use
'd? In line with Luke's comment, it seems it should probably be one or the other?
Here wasi: is listed as a "namespace", the name of the package is http, and the version is specified as 0.1.
If semver is supported, I assume you mean semver + range shortened subsets then? That seems fine and well-defined to me, although a nit is that I wouldn't expect 1.0-alpha1+test
to be valid in this context, or was that a typo for 1.0.0-alpha1+test
?
Finally this can also be used to import multiple versions of the same package:
// wit/deps/http1/types.wit
package wasi:http@1.0.0
interface types {
// ..
}
// wit/deps/http2/types.wit
package wasi:http@2.0.0
interface types {
// ..
}
// wit/foo.wit
use wasi:http/types@1.0.0 as types1
use wasi:http/types@2.0.0 as types2
interface bar {
use types1.{...}
use types2.{...}
}
When use
applies with renaming, I assume the final output component have no traces left of types1
and types2
and instead just reference imports to wasi:http/types@1.0.0
directly? That is, if another package then imports from the above package (say it had package foo
at the top), then it would inherit imports to wasi:http/types@1.0.0
and wasi:http/types@2.0.0
right? And the names used for those inherited imports wouldn't matter from a conflict perspective?
From a transitive perspective the other issue here is that for a given interface, it can have multiple versioned constraints pointing to it from its importer component packages, so the final combined component would either need to have multiple imports for the different constraints (in case they have different resolutions) or it would need to have some kind of constraint intersecting (eg @1.2.3
and 1.2
union as @1.2.3
). There's then in that case the issue of what if you wanted to customize the resolutions for each component differently than the intersection.
Overall, being able to go from an interface to its package, version constraints and internal ID is very much needed at this point and I like the overall shape of what is being proposed. It would be great work through the version management details in a little more detail with regards to how it remains flexible to the standard expected features of being able to control resolutions with lockfiles and custom resolutions while stills getting a good default story, avoiding the various transitive knots.
Seems like a great proposal. My suggestion is to add the dependency manager as part of that instead of having a separate configuration file (https://github.com/bytecodealliance/wit-deps), currently WIT adds complexity with another configuration type so I think it will be better to eliminate extra configuration files. For example the syntax can be:
use foo {
local: ../foo.wit
}
Or:
use foo {
git: https://github.com/example/foo,
path: wit/foo.wit
}
To my option, having everything is one place is easier to read and understand.
then it seems like it'll lead to irregular use and so maybe we should either make it mandatory or disallowed.
A good point! One reason we may want to keep it around though is that if you have your own interface types
and you're additionally importing wasi:http/types
then somehow that'll need disambiguating. If use types
was disallowed at the type then you're forced to rename wasi:http/types
which instead seems like it should be subjectively up to the developer instead?
I do think it's reasonable to largely not mention use foo
syntax for sibling interfaces and instead just document that it works, and then something in like reference documentation you can realize that's how it works and probably continue to ignore it from then on.
... and also easily distinguish indicate where to use the grammar of interface IDs vs just kebab-names. Does that make sense?
Makes sense to me!
If we have packages with multiple .wit files, is the expectation that every single file would have a package signifier at the top?
@guybedford and I talked about this and the conclusion was that to handle this wit-parser
will require at least one *.wit
file in a package to have a package ...
declaration, but it won't require it on all of them. If multiple declarations are specified then they must all match. That way conventionally packages could, for example, have wit/main.wit
which has a package ...
at the top and nothing else needs it at the top.
This seems like a nice simplification for now. Is the assumption then that duplicate names would just throw?
Yes the expectation is that duplication is a parse-error problem. Parsing will always parse all the *.wit
files and then error-out if there's a duplicate interface
or world
amongst them.
If semver is supported, I assume you mean semver + range shortened subsets then?
Guy and I talked a bit about this, but this is definitley a point where the proposal here is a bit shaky. The current intention is that the versions here are glossed over during parsing, perhaps with light validation but not an actually semver resolver/querier/etc. My hope is that everything to do with semver "for real" can be in a separate tool like warg
and the parsing phase can be sort of a "dumb black box" which only does string comparisons of versions. I'm not sure how well this will play out though.
I assume the final output component have no traces left of types1 and types2 and instead just reference imports to wasi:http/types@1.0.0 directly? ... And the names used for those inherited imports wouldn't matter from a conflict perspective?
Correct
My suggestion is to add the dependency manager as part of that instead of having a separate configuration file
To me at least personally I don't think that this is necessarily the way to go at this time. I think there's more nuance than on the surface of how this is handled. My hope is that WIT can be managed in a way similar to Cargo dependencies. The Rust compiler, for example, knows nothing of crate versions. The WIT syntax is sort of like the Rust compiler in that regard where it supports versioned things but it doesn't have its own intrinsic knowledge. Instead an external tool like Cargo is the one managing versions and everything like that (including version ranges, unifying within one semver-compatible range, etc). My hope is that warg
serves this purpose for WIT and the wit-parser
phase of the syntax doesn't have to be too intertwined.
Now that doesn't mean that this sort of information can't be written down in the source. If there's a defined way for wit-parser
to find the result of resolution that seems fine to me too. In the absence of having this already designed, however, it's not something I'd like to take on designing at this point.
I talking with @guybedford a good deal yesterday about all this and these are some of the ideas and results we had:
externname
The current changes to the binary format I outlined above are not sufficient. At the very least instantiation's with
arguments and alias
would also need to be updated with an externname
field. Before doing this, however, one thing Guy proposed was what if instead externname
was only present on imports and with
arguments? Chiefly for example exports would not use externname
and instead would always use kebab-name.
To me this I think makes sense to do since IDs seem mostly relevant for imports and less so for exports. This is actually already codified with world elaboration where if an exported interface transitively relies on an interface that's not otherwise exported then it's implicitly imported. This means that IDs are necessary to avoid conflicts due to implicit imports, but exports can always be explicitly listed with kebab-names in worlds without fear of conflict (as no exports are ever injected).
One concern I would have, however, is the round-trippable-ness of WIT where if you have a world of the form:
world foo {
export my-handler: wasi:http/handler
}
If at the binary level my-handler
is the kebab-name how is wasi:http/handler
inferred? An idea for this is something along the lines of:
(component
(import (interface "wasi:http/handler") (type $t (instance ...)))
(export "my-handler" (instance $my_handler) (instance (type $t)))
)
where here the ID wasi:http/handler
is registered with a type import. With type ascription it's now asserted that the my-handler
export has the expected type here.
So the rough idea here is along the lines of:
externname
only in imports and with
instantiation argumentsOne unknown I realize writing this up is:
world foo {
import wasi:http/handler
export foo: wasi:http/handler
}
That both needs to import the type for wasi:http/handler
and an actual concrete instance, so I don't know how to disambiguate that.
One concern that came up with @guybedford as well is that there's no easy way to import an interface and then describe a world that exports that interface inherently. For example this world:
world foo {
import wasi:http/handler
export foo: wasi:http/handler
}
both imports and exports wasi:http/handler
, so in theory this could be wired up to itself. The instance of a component implementing this world, however, does not itself satisfy the import. Instead this world describes an exported instance which satisfies the import. One hypothetical feature could be:
world foo {
import wasi:http/handler
implements wasi:http/handler
}
(or something like that) where an instantiation of a component implementing this world can be directly wired up to the import. We didn't have any particular concrete conclusions about this per se other than that it might be nice to have at some point (but @guybedford correct me here if I'm wrong)
Hey @alexcrichton , this is a great proposal and reasonably solves many problems that we are facing today with wit packages! Here is my feedback
I like the idea of a top-level declaration of package like package wasi:http@0.1
but I think we should de-couple wit syntax from versions and put the semver of a wit-package outside of the .wit
files. By removing versions from .wit
files, we can enable
.wit
filesI like how rust source files don't care about their version or versions of the dependencies, because the knowledge has completely off-loaded to cargo
. I wish there could be manifest files to a wit package similar to Cargo.toml
and Cargo.lock
that declares a wit package, its metadata, its deps and resolved deps.
Another problem with the deps/* folder syntax is that packages are named after their directory names meaning that there's no way to have two packages of the same name.
Adding a manifest file like Cargo.toml
will be great to help to solve the above problem, as the manifest file can specify the same package twice with different names and different resolution logic.
the conclusion was that to handle this wit-parser will require at least one *.wit file in a package to have a package ... declaration, but it won't require it on all of them.
One of the consequences of not requiring every *.wit file in a package to declare package ...
on the top of the file is that the concept of a package now ties to a filesystem directory. This is because, in a folder, if one file has a package declaration, but other files don't, wit-parser
would treat the entire folder as a single package. Maybe this is what we want, but it often causes confusions. Does the wit package automatically includes all the wit files in the sub-directories as well? Can we have a sub-directory to be a different wit package?
My main concern with that would be redundancy. Bumping the version might mean changing the version in many different files.
If we remove the version from the package declaration, then the package name could stay relatively static. Maybe this could resolve the concern of redundancy by @guybedford ?
I want to bring something that hasn't been implemented yet but relevent to this discussion. I proposed union of worlds a few months ago and I am working in-progress on implementing the include
syntax for worlds.
This proposal extends the path
syntax to not only bring in interfaces
from files, wit package and deps, but also worlds
. But this also introduces ambiguities. There are two problems
The proposed use
syntax can be decalred on the top-level of the file to pull "intefaces" in scope, and allow interfaces and worlds to use a shorter path. So it is effectively functioning as a way to shorten the path when used on top-level. This leads to a quesiton of whether or not it can be used to pull "worlds" in scope. For example, Can a top-level declaration ofuse wasi:keyvalue/world:1.0.1
allow a world to include world
?
Since non-top-level use
only works with interfaces, not worlds, it may make it less clear of what an id of a wit package is referring to. For example, does wasi:http/types@1.0.0
refer to a world or an interface?
Thanks for the feedback!
With respect to versions-in-files, I don't disagree with you. One of the constraints @lukewagner and I were talking about though (sorry forgot to fit this in above) is that if you're presented with a binary component that has two same-ID interface imports that only differ by version then the WIT extraction process needs to present this information somehow. Basically if components allow for differing IDs purely by version then when WIT is extracted the WIT needs to reflect that. We couldn't think of a better idea than putting the version in the WIT file. That being said I very much agree with you that the version is sort of a higher-level concern than parsing. That's why I was trying to keep it as sort of a black-box which is only used for exact string matches at the parsing layer.
For package ..
declarations, I'll admit that I don't have much experience with languages that use this pattern. The example I can think of is Go where files in the same directory all start with package foo
(or at least so I think). The intention for WIT is that subdirectories are not traversed, ever. There's only one way to organize a package and it's files within the same directory. Two packages can't be in the same directory as well. Do you know of pitfalls with this sort of strategy though? (another possibility would be to require package ...
at the top of all files but only require the version on one of them, or otherwise allow the version to appear on any of them)
As for interfaces and worlds I would expect that, yes, worlds could be imported with all the same syntax. Whether the resolved "thing" is an interface or world I figured would be a resolution-level (basically a parse error). Do you think that there's going to be a problem with this though?
Thanks for your reply. Most of what you said makes a ton of sense to me. I need to do more research with respect to the package ..
declarations and will get back to you later. But to the third point -
Do you think that there's going to be a problem with this though?
One concern I have is that it introduces a slight inconsistency. For example, assume that include
has implemented, now users can do
// wit/bar.wit
use foo
interface bar {
...
}
This gives users a sense of feeling that use
as a keyword can be applied to both interfaces and worlds. This is true for top-level use
statement, but it breaks when used inside of a world.
use another-world // ok
world bar {
use another-world // error
}
Well, we know that the correct way to include a world to world bar
is to use include
. The main point is that the two use
s, one from the top-level, and one within interfaces and worlds, are two different things that have different semantics, but we are calling the same name.
Altihough I am probably suggesting to seperate these two to two different keywords, I don't have a good name to replace the top-level use
on top of my mind. import
is a popular one, but it's reserved already.
Alternatively, since include
proposal is still WIP, we can think about if it's feasible and make sense to just use use
keyword to union worlds.
If I understand right, with or without include
this is still an issue? With this proposal you could use foo
to import an interface at the top level and then within an interface you do use foo.{...}
, or in other words a bare use foo
is disallowed from within an interface or a world. This means that use
, the same keyword, is acting differently in two locations?
One possibility is to scrap top-level use
anyway as it's somewhat "just" sugar since the full forms are allowed within interfaces/worlds. I agree it's a bit odd to use the same keyword for different things.
Ok I talked some more with Guy and Luke today, and here's some conclusions we reached:
We decided that one rule of thumb we'll stick to for exports and imports is that there's a "canonical string" by which they can be referred to. This means that while we may add more structured forms or more structured strings there it will always be possible to have a string key.
This means that (with "..."
and (alias export $C "..."
don't need to change for example as the strings will still work there. What will change, however, is the import
and export
statements, namely:
(component
(import "foo" (func)) ;; today's syntax
(import (interface "wasi:foo/bar@2.0") (instance ...)) ;; proposed syntax
(export (interface "...") ...) ;; proposed syntax
)
Specifically just imports and exports will get the externname
changes. Textually they'll be separated with (interface ...)
and in the binary format we'll switch to using prefixed payloads:
externname ::= 0x00 n:<name // kebab-name
| 0x01 n:<name> // interface
where the syntax for a string accepted for an interface
is:
interface ::= ( namespace:<kebab-name> ':' )? package ( '@' version )?
package ::= pkg:<kebab-name> '/' interface:<kebab-name>
version ::= [1-9][0-9]* '.' [0-9]+
Given the above binary format, it's not actually possible to have both a kebab-name and an ID for an import or export. This means that this will be disallowed:
world foo {
import wasi-http: wasi:http/handler@2.0 // ERROR: both a kebab-name and interface specified
}
however this will still work:
world foo {
import my-interface: interface { /* ... */ }
}
basically kebab-names will only be used for locally defined things rather than interfaces. Sort of a "one-off developer mode". Automatic injection of dependencies will never insert kebab-names, only IDs.
Ok that was actually a shorter comment than I thought it was going to be! (correct me if I missed anything @lukewagner or @guybedford)
Thanks again for the clear write-up!
One small tweak I'd suggest is that, given that we're saying that "one-off developer mode" is meant to use kebab names, not <interface>
names, and thus with <interface>
names we're in a registry-ish setting, perhaps the namespace
should be non-optional, at least until we have concrete use cases where we definitely don't want one.
The only other thing I'd like to clarify is that, since today name
is defined (in the text and binary format) to be an annotated kebab-name:
interface
case of externname
would contain an <interface>
, not a <name>
with
and alias
would need to be generalized in-place from <name>
to <string>
(i.e., a plain UTF-8 string, just like <core:name>
) to be able to match either <name>
or <interface>
Makes sense to me to require namespace
for now yeah, we can always relax it later if need be. Otherwise yes you're right on <interface>
vs <name>
vs <string>
, sorry I was a bit sloppy in my notation above :)
I've been talking with @lukewagner and @guybedford recently about #177 and how best to tackle that in components and WIT, and I'm opening this issue to propose changes to both WIT and the component model binary format to solve the issue at hand and a few neighboring concerns as well. As a heads up what I'm proposing here is a breaking change relative to the currently implemented and accepted WIT syntax, and it's an addition of the binary syntax (while adding some ignores of some preexisting binary syntax).
Motivation and Existing Problems
The main motivation of these changes is that if you have a world such as:
then any dependency of
foo
will be "inlined" into the world as well:This interacts poorly with transitive dependencies where, for example the
proxy
world for WASI depends on atypes
interface in thewasi-http
package so the world ends up havingimport types: ...
which is quite a "bland" name without any context in it.One way to fix this is to consider an "id" more often than a kebab-name. For example the "id" of the relevant interface is
wasi:http/types
which is a much more contextually relevant name. The problem with this, however, is that WIT doesn't have a great means by which to discover its own package name or namespace. For example when developing thewasi-http
package itself you'd simply have:So there's no context available to say wither
wasi
orhttp
. This is sort of solve with thedeps
folder thewit-parser
crate implements along the lines of:where here the parser at least knows the package name is
http
. Still, however, it doesn't know anything aboutwasi
.Another problem with the
deps/*
folder syntax is that packages are named after their directory names meaning that there's no way to have two packages of the same name. For example two packages from different registries or two packages from different versions of the same registry.In general many of the above problems are solvable in isolation somewhat but I'm hoping to solve them all at once with updates to the WIT syntax, binary format, and translation to wasm. One final point worth addressing here, or at least planning to be addressed here, is at least somewhat introducing the concept of versioning for future use. This isn't intended to be comprehensive just yet, but should help set up some foundations hopefully.
Changes to WIT
There are a number of syntactical changes to WIT described here and this attempts to go through them in such an order to build up to a final picture at the end.
Package names in files
The first change I'd like to propose to WIT is a requirement that all
*.wit
files will start at the top with apackage ...
statement. For example you could write:This will solve the "what's the name of my package" problem from above by now it's explicitly declared inline. This is additionally where we can have other semantic information such as versioning. An example of
wasi-http
might be:Here
wasi:
is listed as a "namespace", the name of the package ishttp
, and the version is specified as0.1
.Valid package names will be:
kebab-name
- I'm just doing somethingkebab-name@1.0
- I'm doing something slightly more seriouskebab-namespace:kebab-name
- I'm developing for a registrykebab-namespace:kebab-name@1.0
- I'm very serious at this point (most WASI repos will probably look like this)For now the version could probably be any semver (e.g. up to
1.0-alpha1+test
or something like that.The
package ...
header would be required at the top of all*.wit
files that make up a package. Eachpackage ...
directive would also be required to match all the other one specified, including the version.Package Organization
WIT today is structured as a
Package
has a set ofDocument
s where each document is a file (typically*.wit
). EachDocument
can have a set of interfaces and worlds, where at most one world and one interface can be flagged asdefault
. I'm proposing to change this instead to:Package
is a collection of interfaces and worlds.*.wit
files but the interfaces/worlds in a package are unioned together. This means you can't defineinterface foo
in two files in the same package. Packages can always be represented with one file.default
keyword goes away entirely.These changes are intended to assist with being able to auto-assign a unique ID to any interface within a package. They will also affect
use
paths described below. When combined with the above change topackage ...
headers some examples of inferred IDs are:The "ID" is inferred to be the package name, without the version plus a slash, plus the name of the world or interface. The version of the package additionally becomes the version of the interface.
The rationale for removing
default
is more clear with the changes touse
below, and otherwise all WIT files being merged together means that for allinterface
andworld
items there's a unique ID within the package.Changes to
use
Today the
use
statement looks like:With the updates to package structure above these forms largely no longer make sense. Instead this proposes making IDs more first-class in WIT syntax. The
use
statement will still be somewhat similar grammatically where it will look like:where the major change is how an
interface
is specified. To explain that here's a few examples. First is importing from another interface in the same file:Here the
interface
is just the bare namefoo
as that's what's in scope. Iffoo
where in separate file then it needs to be explicitly imported in the outer scope to be used.Here a bare
use foo
at the top level pulls the interfacefoo
into scope. Due to the name beingfoo
, that means "find an interface namedfoo
elsewhere in this package. In this situation it's found inwit/foo.wit
.Importing from a foreign dependency now happens through the ID of that dependency. For example, with a
deps
structure, it would look like:Here the
use the-dependency/foo
statement will look for a package with the namethe-dependency
which is found within thedeps
folder in this case. Here the namethe-dependency
matches thepackage
declaration. Forwasi-http
this would look like:Here it can be seen that
wasi:
namespacing syntax is allowed as well. Furthermore it can be seen that the name of the folder holding thewasi-http
package,http
, does not have to exactly match the package specifierwasi:http@1.0.0
, it can be named anything.This leads us to an example where you can depend on two packages of the same name from different registries:
The
as
syntax will be allowed to rename dependencies locally within a file's context to avoid name collisions. The inferred ID of thewasi-types
andba-types
interfaces will still be unique as they're derived from unique package IDs.Finally this can also be used to import multiple versions of the same package:
The
@version
syntax is allowed inuse
statements to disambiguate which version is being imported if multiple exist. It's an error for there two be two candidates without a version specified, for example. If only one version is available it's inferred to be that version (as is the case for all above examples).Finally, all of the above can be slightly more "sugary" as well. The top-level
use
can also be used inline within worlds and interfaces. For example this:is equivalent to:
This is provided when something is only needed in one location to avoid the top level
use
and assigning a name to it. Note, though, that this can get sort of sigil-heavy withuse wasi:http/types@1.0.0.{...}
so it's not intended to be super widespread. That being said these two are also equivalent:is equivalent to:
so the
use foo
statements to import package-sibling interfaces is not required.Changes to
world
Currently imports and exports to a
world
are of the form:This suffers from the downside mentioned originally where each interface must be assigned a kebab-name. The new syntax for worlds will look like:
This means that the
kebab-name
is now optional. It'll be required for unnamed interfaces such asimport foo: interface { ... }
but otherwise it's now possible to do:Here there's no kebab-name. This means that the "name" of the import will be the ID of the interface, which in this case is
wasi:http/types
(as imported from a siblingwit/types.wit
. A kebab-name can be explicitly listed, but often won't be required any more.For example a Fastly-specific world might look like:
Here no kebab-names are necessary and everything, including their transitive dependencies, will be assigned names based on IDs. Transitive dependencies will always be inferred to be named by their identifier to avoid clashes between transitive dependencies.
Changes to the binary format
Currently the binary format has:
This proposes changing this to:
Here the 0x00 production is still interpreted as "here's a kebab name for the thing". The 0x01 production is reinterpreted to ignore the URL for now (for lack of a better idea of what to do with it). This could perhaps get removed during a final 1.0 break. The 0x02 production is the ID-based form of import which refers to "I'm importing an interface". I know @lukewagner is also interested in perhaps an 0x03 form of import meaning "I'm importing a specific component implementation" with perhaps more options too, but I don't think that affects this proposal in particular.
A text format for these could perhaps look like:
I think that's everything I wanted to cover for this, and I'm interested in hearing feedback from others using WIT who have thoughts on syntax, bikeshedding, the breakage involved here, etc.