search

uutils / coreutils

Cross-platform Rust rewrite of the GNU coreutils
https://uutils.github.io/
MIT License
17.74k stars 1.27k forks source link

WASI support #4475

Open connor4312 opened 1 year ago

connor4312 commented 1 year ago

I've made a fork of coreutils that implements provisional support for running most coreutils under WASI.

Currently this involves quite a bit of conditional flagging, in large part because Rust hasn't adopted newer proposals that implement file permissions to WASI. However, the number of changes is not terrible, though there are still quite a few warnings I'd need to fix up.

I was wondering what the appetite is of coreutils maintainers to merge something like this into the repo and start officially supporting WASI, at least for a subset of tools.

tertsdiepraam commented 1 year ago

That's very cool!

Not sure how the other maintainers feel, but I'm in favor of merging something like that! It would be another step towards making uutils truly "universal" :)

The number of differences with other platforms indeed feels manageable. In order to make the support "official" we will have to add it to the CI though, so we don't break it by accident.

at least for a subset of tools.

Do you know what subset of utils is useful on WASI? Do unix-specific utilities like chmod make sense? I think the first version we could merge would require support for at least the "common core" set of utils (the ones enabled by default).

Edit: Why are you requiring path dependencies for libc and nix? Do you need to patch them to work with WASI?

connor4312 commented 1 year ago

Great! 🙂 This is still a rough sketch and there's polishing to do before we'd be ready to PR. I don't know if that'll happen this month or next.

Also cc @dbaeumer who is leading the wasi work for VS Code

Do you know what subset of utils is useful on WASI? Do unix-specific utilities like chmod make sense? I think the first version we could merge would require support for at least the "common core" set of utils (the ones enabled by default).

chmod would be relevant when file permissions are available in Rust's wasi libc. I think this is even possible today if one uses the wasi-sdk, though I have not tried it, and imo we should not write code that doesn't work with Rust/clang out of the box.

However, not all utils reference functionality present in WASI proposals. For example, there are no processes for kill to kill.

I agree that targeting a common core set of utils makes sense.

Edit: Why are you requiring path dependencies for libc and nix? Do you need to patch them to work with WASI?

Yea, I have a couple of PR's into both nix and libc at the moment (https://github.com/rust-lang/libc/pull/3142 https://github.com/rust-lang/libc/pull/3143 https://github.com/nix-rust/nix/pull/2011)

sylvestre commented 1 year ago

Not sure how the other maintainers feel, but I'm in favor of merging something like that! It would be another step towards making uutils truly "universal" :)

+1 :)

dbaeumer commented 1 year ago

Our current goal is to implement a WASM/WASI host for VS Code running in the browser so that web assemblies have transparent access to the VS Code file system to the console and the terminal. Here is a small demo running CPython in the browser with access to the files in the workspace which are hosted in a GitHub repository:

cast

What we would like to provide as well is some sort of basic terminal support so that users can execute python directly or run command like ls, cat, ...

This being said commands like kill or chmod are not high priority for that support.

tertsdiepraam commented 1 year ago

That sounds awesome! I'd love for uutils to be a part of that effort!

RReverser commented 1 year ago

FWIW I had a WASI-enabled fork of coreutils a while back as well, with my custom JS runtime to have coreutils working in a browser: https://wasi.rreverser.com/ (https://github.com/GoogleChromeLabs/wasi-fs-access, fork here https://github.com/RReverser/coreutils)

It allows to mount even with an actual filesystem in Chromium-based browsers! (video demo here https://www.youtube.com/watch?v=qRmO-8b4WmE)

Unfortunately, I never got around to upstreaming my changes and they're probably more outdated than @connor4312's, just thought I'd post for posterity :)

chadbrewbaker commented 1 year ago

@connor4312 I have some time to write a GitHub worker YAML. Sorry, I didn't see in your code. Were you using Chromium or some other WASM/WASI runtime? Also what is the list of WASM/WASM runtimes that we are expecting to support?

connor4312 commented 1 year ago

I was using wasmer to develop locally. But out of the box rust targets a very minimal set of WASI APIs so I would expect it to be quite portable. Dirk was able to build upon this work to run the subset of coreutils in Chrome.