Filesystem path inference in Preview2

[sunfishcode]

@npmccallum @ricochet

Following up on the discussion of preopen paths in the meeting today, here's a more complete discussion.

A high-level point is that none of these details would be baked into wasm modules. Some host environments won't have filesystems, and some won't have command-line interfaces, but they can still present preopens to command-profile wasm binaries. WASI won't require that hosts follow any specific UI or conventions here. The main reason for bringing this topic up is that with wasi_preview_snapshot1, the --dir option ended up being fairly widely implemented, so there is a desire for a common convention, even if it isn't required.

This also means that engines could also add their own customizations. There could be a "prompt the user" option on top of the inference scheme, or a "--dir=" layer that works as a filter for what paths are allowed to be inferred, or other things. Many environments have unique security concerns, and we don't need to address all of them here.

With that said, I'll sketch out two alternatives, to start off the discussion:

Option A: Infer paths

This is what I described out in the presentation.

$ grep red green/blue.txt %=not/a/path

Briefly:

• A leading % is special:
  • A leading %= means to pass the rest of the string as a string and not a path.
  • Other % prefixes could be defined.
• Everything containing a / is a path.
  • Optionally with refinements to recognize idioms like --input=foo/bar where the path is a substring, or colon-separated lists of paths.

Pros: Convenience. Most users can just type paths most of the time, and it'll often work. The most common problem will be current-directory paths with no slash, and users will need to add a ./. We can give error messages that can guide them to this though.

Cons: The Bobby Tables problem. If an attacker can influence a string passed as an argument to contain a slash, it can cause the program to open an unintended file. It's not as severe as it may seem though, because if the program isn't expecting a path, then it won't attempt to call open on the string. And if the program is expecting a path, then it's already unwise for users to be passing untrusted path strings into it. Admittedly, it isn't absolutely waterproof, but it might be a reasonable default tradeoff in convenience vs. usability.

Option B: Explicit paths

Another option would be to have an explicit syntax for paths, something like braces:

grep red {green/blue.txt} not/a/path

Anything in braces is a path. There are many possible syntactic variations here. Maybe double braces like {{this}} would be better. In any case:

Pros: It's a simpler system with fewer special-case rules. Users wouldn't need to know about the ./ trick or %= literals or other things. And it's more resistant to Bobby Tables, though still not perfect.

Cons: Convenience and portability. Users would need to type extra for every file. And they'd need to pass in paths differently than they do on other platforms.

Others?

[abrown]

For completeness, option C would be to specifically ask the user, as Syrus proposed. This could look like prompts on the CLI, e.g., or graphical prompts (I believe this is how @sunfishcode interpreted things in the meeting). I think Syrus' intention was to reference Deno's runtime prompts (see the first paragraph of the permission docs). The downside to prompts is that they are annoying, as implied by this kind of article.

---

I am not actually advocating for option C above, just recording it. I am a bit confused why, if we have the types of the command available in WIT, we can't use that information to "know" which arguments were paths and which were not. I suspect it is because there is no "path" type available, but if there were then maybe, hazily, this "use the WIT types" idea could eventually become option D?

[npmccallum]

@sunfishcode Isn't this an implementation detail? I mean, I'm not sure it is actually relevant to the WASI group. I would expect all WASM runtimes to export an API for setting up everything manually without automatic magic. This strikes me as a CLI concern.

[ricochet]

To add to @abrown's thoughts, I also would rather jump past what was described as level 0 in the presentation (args are typed list<string>) to the later Wit-based options. This is strongly preferred over risking a potentially vulnerable abstraction.

[sunfishcode]

For completeness, option C would be to specifically ask the user, as Syrus proposed. This could look like prompts on the CLI, e.g., or graphical prompts (I believe this is how @sunfishcode interpreted things in the meeting). I think Syrus' intention was to reference Deno's runtime prompts (see the first paragraph of the permission docs). The downside to prompts is that they are annoying, as implied by this kind of article.

I agree that prompts may make sense in some contexts; as I mentioned above, they could be added as a layer on top of this system. In wasi_snapshot_preview1, a lot of engines implemented --dir, as it was useful to have different engines work in the same way, so I'm seeking a convention which could serve a similar purpose for Preview2.

I am not actually advocating for option C above, just recording it. I am a bit confused why, if we have the types of the command available in WIT, we can't use that information to "know" which arguments were paths and which were not. I suspect it is because there is no "path" type available, but if there were then maybe, hazily, this "use the WIT types" idea could eventually become option D?

The reason we can't do this yet is that it depends on what I've called Levels 1 or 2 of the Typed Main plan, which is where users write their own Wit files for their commands. To keep Preview2 scoped, we're currently sticking to Level 0 for now, which is the level where users don't write a wit file and there's still just a fixed argv: list<string>. But in the future, I hope we'll be able to have real typed signatures, which, as you say, will allow us to eliminate a lot of the guesswork here.

@sunfishcode Isn't this an implementation detail? I mean, I'm not sure it is actually relevant to the WASI group. I would expect all WASM runtimes to export an API for setting up everything manually without automatic magic. This strikes me as a CLI concern.

I agree. I aimed to address this in the paragraph above about how "none of these details would be baked into wasm modules".

[sunfishcode]

To add to @abrown's thoughts, I also would rather jump past what was described as level 0 in the presentation (args are typed list<string>) to the later Wit-based options. This is strongly preferred over risking a potentially vulnerable abstraction.

My only reason for not including level 2 in Preview2 is just to scope the milestone to the minimal amount of work needed to get WASI rebased on Wit, because that's a specific thing that a lot of other things are blocked on. However, I'm eager as anyone about Typed Main, and wouldn't object if people wanted to work towards making that happen :smile: .

[npmccallum]

@sunfishcode If this is an implementation detail, then I would suggest the WASI group is the wrong audience for it.

[lukewagner]

Agreed that this is primarily a host/embedder detail: when a cli/Command component is called and given a list<(string,preopen)>, the component doesn't know how this list was created (or, in the component model, whether the caller is the host at all). Thus, you could imagine 3 different hosts all running the same component but using 3 different ways of conjuring up the list<(string,preopen)>. In this regard, a host sortof plays a role symmetric to "user agents" on the Web. That being said, even though each host is free to do its own thing, I think there is still some value in having discussions on these "non-normative" details in the WASI setting so that, when a host does in fact want to run a cli/Command in a traditional Unix-style command line (several engines do this now, and somewhat-compatibly), there is some well-reasoned guidance.

Separately: it's a great question of whether it's worth defining such an "untyped" profile as cli/Command. Taking the same argument farther, the same question could be raised even for filesystems. But here's the reason why I think it's valuable: today, there are huge swaths of already-written code that roughly assume the cli/Command profile (or could be "easily" adapted to it, as we've seen with all the folks adapting to wasi_snapshot_preview1 already) and, even once the component model and wit are 100% feature complete, we'll still have to ask how we can reuse this existing code in this wonderful new declaratively-typed component world.

A default answer might be: they can manually port to it. But I think a more attractive, incrementally-adoptable answer could be this:

1. M different language toolchains target preview2 (the cli/Command profile). The concepts defined by cli/Command allow a lot of their existing standard library stuff to just work, and so a bunch of existing code can just work. Initially, these modules only run on CLI hosts and thus not on, e.g., an HTTP Proxy (which doesn't want to call untyped-main or supply a filesystem).
2. In parallel, we define N more profiles in WASI (e.g., http/Proxy) specialized to the various non-CLI host environments we're all working on. These other profiles would have typed exports and no filesystem (at least on hosts that have no native filesystem (e.g., http/Proxy)).
3. We put the linking/virtualization features of the Component Model to real use, creating a build tool that takes as input a cli/Command component and spits out a new component wrapping the cli/Command and targeting a different profile (e.g., http/Proxy) by synthesizing a small virtual filesystem and typed-to-untyped-main trampoline in the wrapper component. This is roughly what Emscripten does and it's very effective; we'd just be doing it using the component model instead of JS glue code.

The cool thing about this approach, I think, is that it lets the M toolchains in 1 and the N hosts in 2 share the virtualizer tool in 3. It also allows the ecosystem to pool its efforts in developing a featureful, robust virtualizer, instead of the MxN different projects doing this in their own bespoke way. Of course, once new code is written to start using typed mains and declarative imports using wit, they're not targeting cli/Command and not using the virtualizer. But this approach helps us present a pay-as-you-go model to developers for incremental adoption. It also helps hide some of the latency, Tomasulo-style ;)

[yamt]

$ grep red green/blue.txt %=not/a/path

i'm not sure if i understand the context.

• who is prompting with $?
• what's grep here?

does it somehow imply something like the following?

$ wasmtime run grep.wasm -- red green/blue.txt %=not/a/path

[sunfishcode]

The $ in the example represents the user's shell, however it's not doing any of the work in this example. The grep here represents a grep-like program compiled to WASI.

Yes, it represents that wasmtime command line. One of the goals here is to have a command-line syntax that isn't specific to one wasm engine, and doesn't require engine-specific out-of-band options to use.

[yamt]

The $ in the example represents the user's shell, however it's not doing any of the work in this example. The grep here represents a grep-like program compiled to WASI.

Yes, it represents that wasmtime command line. One of the goals here is to have a command-line syntax that isn't specific to one wasm engine, and doesn't require engine-specific out-of-band options to use.

ok. thank you for explanation.

my understanding is:

• there is no standard way to parse cli arguments (even if we assume getopt, it isn't aware of paths.)
• thus, argument parsing needs to be done inside the wasm module
• when the wasm module parsed the arguments, it's too late to specify --dir.

random options i can think of:

• invent our own standard of cli argument parsing. (so that it can be done by the runtime.) if we want to deal with non-positional arguments (eg. -o filename), something richer than getopt is necessary. (is it in the scope of "typed main"?)
• somehow let the wasm module to tell the runtime about paths after parsing arguments. eg. a dedicated entry point to just parse arguments.

Option B: Explicit paths

if we need explicit annotations, i suspect it's simpler to standardize some of the runtime settings and provide a way to pass them. eg.

grep {--dir .} red green/blue.txt not/a/path

after all, it might not be only about paths. eg.

telnet {--addr-pool xxx} xxx

[sunfishcode]

Yes, in the limit, the full Typed Main concept means that the Wasm engine is doing all the command-line parsing. The details of how command-line parsing is done, whether it's Unix-style with -o or Windows-style with /o, or even things that aren't even command lines at all, will all be up to the host environment. We may still find it useful to define a standard parsing convention, but it won't be baked into the wasm.

What I'm proposing here for Preview2 is a simplified version of it that doesn't require as many changes to existing code and existing toolchains. It still leaves most of the command-line arg parsing inside the wasm module, but it does preprocess a few things out.

I also agree that there needs to be ways to pass things other than files and directories. My proposal above has % as a reserved prefix which could be used for this. Extending the Option B {...} as you say is also an option.

[sunfishcode]

This is still an interesting idea, but it's not currently being pursued.