ssokolow / nodo

Pre-emptively created repository so the design can be discussed on the issue tracker before commits are made (repo name may change)
Apache License 2.0
18 stars 0 forks source link

Decide on a design for a minimum viable v0.1 #1

Open ssokolow opened 2 years ago

ssokolow commented 2 years ago

As a "Just Works™ baseline sandboxing wrapper for any build automation", what features does this project need for a minimum viable product, and what features should be agreed on as definitely suitable for a later release?

Current ideas for the design:

So far I'm thinking:

...and a configuration file design which provides this sort of functionality, but not necessarily with a schema anything like this:

# Default list of root-relative paths to be denied access to
# (The idea being to provide an analogue to `chattr +a foo.log`
# so `git diff` can be used to reveal shenanigans)
blacklist=[".git", ".hg", ".bzr", ".svn"]

[profile.cargo]
root_marked_by=["Cargo.toml"]
root_find_outermost=true  # For workspaces
projectless_subcommands=["init", "new"] # Assume $PWD is project root
allow_dbus_subcommands=["run"]
allow_gui_subcommands=["run"]
# allow_network=false
allow_network_subcommands=["add", "audit", "b", "build", "c", "check", "fetch", "run"] # etc. etc. etc.
deny_subcommands=["install", "uninstall"]  # must be run unconstrained

[profile.make]
root_marked_by=["Makefile"]
cwd_to_root=true  # run `make` in project root no matter where `nodo make` is run from

Requirements for v0.1 MVP

v0.1.1

v0.1.x

v2.0

NobodyXu commented 2 years ago

Cargo has a command called cargo locate-project --message-format plain which prints the root of the project to stdout, I think we can use this to find the root of the project.

ssokolow commented 2 years ago

That would involve spawning an additional subprocess on each invocation. Do you have evidence that doing it that way is an improvement over just walking up the filesystem ourselves to find Cargo.toml?

After all, not all build systems that will need to be supported are going to have an equivalent command and some (eg. npm) are horrendously slow to start up, so the machinery to walk up the filesystem looking for a file is going to need to be implemented either way.

UPDATE: ...and it would be an additional point of failure, since the command being invoked to identify the path to sandbox could itself become compromised and can't be run sandboxed.

NobodyXu commented 2 years ago

The problem with manually finding Cargo.toml is workspace.

In that particular situation, finding Cargo.toml is not enough,

Edit:

After finding the first Cargo.toml, you would also have to check whether its parent dirs has Cargo.toml and verify that.

IMHO in the v0.1 MVP we could just invoke an external command.

NobodyXu commented 2 years ago

I also would like the following firejail sandbox options to be applied when building packages:

private-etc alternatives,ca-certificates,ld.so.cache,ld.so.conf,ld.so.conf.d,ld.so.preload,resolv.conf,ssl

private-dev
private-tmp

caps.drop all
nonewprivs # Make setuid binaries has no effect

noroot

dbus-system none
dbus-user none

blacklist /opt
blacklist /media
blacklist /mnt
blacklist /srv
blacklist /sys
blacklist /run

blacklist /boot
blacklist /root
blacklist /sbin
blacklist /snap
blacklist /sys
blacklist /var

noexec /bin/su
noexec /usr/bin/sudo
ssokolow commented 2 years ago

After finding the first Cargo.toml, you would also have to check whether its parent dirs has Cargo.toml and verify that.

That was what the root_find_outermost=true # For workspaces in the draft config file was.

I was already taking the need to find the last match rather than the first into account.

I also would like the following firejail sandbox options to be applied:

I'm pretty paranoid about that, though some of those may have to wait until v1.1 because I'm not sure per-profile Firejail rules will make v1.0 and I suspect that private-etc and some of those blacklists might break some of the non-Cargo build tools I want to support.

NobodyXu commented 2 years ago

That was what the root_find_outermost=true # For workspaces in the draft config file was.

I was already taking the need to find the last match rather than the first into account.

Sounds good to me.

I'm pretty paranoid about that, though some of those may have to wait until v1.1 because I'm not sure per-profile Firejail rules will make v1.0 and I suspect that private-etc and some of those blacklists might break some of the non-Cargo build tools I want to support.

I think that at least some of them should be applied unconditionally:

private-dev
private-tmp

caps.drop all
nonewprivs # Make setuid binaries has no effect

noroot

noexec /bin/su
noexec /usr/bin/sudo

blacklist /sys
blacklist /boot
NobodyXu commented 2 years ago

I would also like to decide the language to be used for this project.

IMO dynamic-typing or weak-typing languages like python and bash is not suitable for security-sensitive application like this since you can easily work around their typing system.

Since I don't have much experience in Go, IMHO using Rust in this project sounds reasonable.

NobodyXu commented 2 years ago

It seems that firejail already has a cargo.profile though IMO it is actually too strict.

ssokolow commented 2 years ago

I think that at least some of them should be applied unconditionally:

I'm certainly hoping to do more than that for v0.1. I'm just not sure we can do all of the longer list until we have support for per-profile overrides.

Since I don't have much experience in Go, IMHO using Rust in this project sounds reasonable.

I thought using Rust was being taken as a given before we even moved off Reddit. The only reason I even mentioned Python is that, at one point, I was considering starting from a Python script I'd written for something else.

Bash is certainly not something I'd want to write something longer than a dozen lines in (I use Python for my quick "shell scripting" and Rust for my more long-term "shell scripting") and I've never used Go because I don't like how primitive it is.

It seems that firejail already has a cargo.profile though IMO it is actually to strict.

I generally only take ready-made Firejail profiles as advice on what I can do in my own from-scratch profiles without causing the program to fail to start.

NobodyXu commented 2 years ago

I'm certainly hoping to do more than that for v0.1. I'm just not sure we can do all of the longer list until we have support for per-profile overrides.

Me too, I also cannot be 100% sure that all of them in the longer list is appliable.

I thought using Rust was being taken as a given before we even moved off Reddit. The only reason I even mentioned Python is that, at one point, I was considering starting from a Python script I'd written for something else.

Bash is certainly not something I'd want to write something longer than a dozen lines in (I use Python for my quick "shell scripting" and Rust for my more long-term "shell scripting") and I've never used Go because I don't like how primitive it is.

Cool.

I generally only take ready-made Firejail profiles as advice on what I can do in my own from-scratch profiles without causing the program to fail to start.

I just thought it can be used as a reference.

ssokolow commented 2 years ago

I just thought it can be used as a reference.

No problem. Thanks for saving me the trouble of going looking.

ssokolow commented 2 years ago

I just thought of another thing that'll need to be configurable. Whether to canonicalize (i.e. resolve symlinks) or just "make absolute" when ascending to find the ancestor in the path which contains root_marked_by.

Some build tools may do one while some may do the other so, to Just Work™ without introducing confusing edge cases that may cause problems, it'll be necessary to have something like root_resolve_symlinks to enable matching what the command to be wrapped does.

(eg. Rust has std::fs::canonicalize but no "make absolute" in std yet while Python calls "just make absolute" os.path.abspath which sounds more inviting and "correct to intention" than os.path.realpath, which is how you canonicalize.)

NobodyXu commented 2 years ago

I just thought of another thing that'll need to be configurable. Whether to canonicalize (i.e. resolve symlinks) or just "make absolute" when ascending to find the ancestor in the path which contains root_marked_by.

Some build tools may do one while some may do the other so, to Just Work™ without introducing confusing edge cases that may cause problems, it'll be necessary to have something like root_resolve_symlinks to enable matching what the command to be wrapped does.

That's indeed another thing that needs to be configured.

ssokolow commented 2 years ago

That said, given that Rust has no "make absolute" in std yet, I'll need to decide how to go about it.

(i.e. For security reasons, it may be a good idea to just do a one-to-one port of the Python code until something for that shows up in the Rust standard library rather than relying on a third-party crate.)

NobodyXu commented 2 years ago

(eg. Rust has std::fs::canonicalize but no "make absolute" in std yet while Python calls "just make absolute" os.path.abspath which sounds more inviting and "correct to intention" than os.path.realpath, which is how you canonicalize.)

According to here, std::fs::canonicalize seems to return an absolute path.

ssokolow commented 2 years ago

It's a wrapper around the libc realpath(3) function. "realpath() expands all symbolic links".

os.path.abspath just performs string manipulation based on $PWD and the given path.

They behave differently in a situation like this:

Canonicalizing will turn ~/src/project1/src/mymod into ~/src/project2/src/mymod before we ascend looking for Cargo.toml while a non-canonicalizing "make absolute" function will not.

NobodyXu commented 2 years ago

Thanks for explaining this to me.

I didn't realize that symlink handling is different.

ssokolow commented 2 years ago

As for how we do it, since there's no "just make absolute" function in std yet, I may want to do a one-to-one port of the code for os.path.abspath from Python and then do some differential fuzzing to make sure that the behaviour is identical to the mature Python implementation.

It's a little too central to the security proposition for me to feel comfortable relying on a third-party crate.

NobodyXu commented 2 years ago

Agreed.

ssokolow commented 2 years ago

Of course, I'll put it in its own module and file so keeping the licenses separate and the ported code easy to replace is as simple as possible.

ssokolow commented 2 years ago

Another thought to consider when I have time to do some research. Maybe I should use a proper argument parser but configure it in "the first non-option implies -- (all following arguments are positional)" mode and support options like --allow-network to make it feasible to do something like "Run nodo --allow-network npx <command> once to install it, then run nodo npx <command> to use it".

(The main question being whether there are other details about how commands like npx function which would make that not the best choice. For example, how does its cache work?)

NobodyXu commented 2 years ago

I am not familiar with npm.

AFAIK about cargo, this seems to be perfectly OK.

ssokolow commented 2 years ago

npx is a command that comes with NPM that's essentially a hybrid of cargo install and cargo run. It'll download the package you specify, cache it to a user-global install location akin to ~/.cargo/bin but not in PATH, and then run the command.

Once WASI's APIs are richer and more crates support targeting it, wax would be a Rust equivalent with built-in sandboxing for anything published to Wasmer's WAPM repository.

In fact, this project is inspired by wax's default sandboxing behaviour. (Because WASI uses capabilities-based security, you have to pass in handles for the pieces of the filesystem you want the program to be able to see. wax defaults to granting $PWD when you run something with it.)

NobodyXu commented 2 years ago

Sounds amazing, though the combination of downloading and run makes sandboxing it harder.

ssokolow commented 2 years ago

You can always use the wapm command to download and run things separately. npx and wax are just wrappers to make things as simple as possible.

Also, WebAssembly has its own sandboxing so, if you trust wax, it can deny the command network access without needing to itself be denied permission to access the network for updates. (Like JavaScript running in a browser with uMatrix.)

In fact, wax is just a wrapper script around wapm execute minus the --offline flag. wapm install <command> with one sandbox profile, then wapm execute --offline <command> with another.

NobodyXu commented 2 years ago

IMHO, it’s probably not necessary to sandbox wasm since it has builtin sandbox via its capacity based API.

So maybe we should focus on cargo and npm instead, since these two doesn’t have any builtin sandbox?

ssokolow commented 2 years ago

With the way I want this to operate, once we've got profiles for Cargo and npx set up, it's just copy-pasting a dozen lines of TOML and editing the command and subcommand names to support WAPM and wax.

The most time-consuming part would be running the tests to Do It Right™ and confirm my suspicion that, being written in Rust, Wasmer's WAPM CLI tool uses std::fs::canonicalize for making paths absolute.

NobodyXu commented 2 years ago

That matches with my impression of this project so far.

On the other hand, where shall we put the configuration file? I think $HOME/.config/nodo/config.toml will be a good place for such config file.

Also, I think we can configuration file splitting v0.1.x so that users can have cargo-profile.toml, npm-profile.toml and etc.

ssokolow commented 2 years ago

Definitely something compliant with the XDG Base Directory Specification, so something starting with ${XDG_CONFIG_HOME:-${HOME}/.config}.

I'll want to think about whether to re-implement that to protect people who install with cargo install without --locked though.

I'm not sure about configuration file splitting though. I've learned to follow YAGNI the hard way and I haven't yet had a solid rationale that convinces me it's worth the added maintenance burden and risk of overlooking a way it could make the design more vulnerable.

That sort of thing is more for when you want your creation to be a piece of infrastructure a project like Cargo or NPM can integrate support for, and that would divide the responsibility for ensuring the configurations are trustworthy compared to just having all the vetted ones in the same repo as the code which enforces them.

...but I will want to think about whether it's worthwhile to let users extend the default config rather than completely overriding it, given that things like adding new profiles would require them to add new aliases, and making the rules more nuanced will probably result in a new schema version for the config file.

(I do plan to have a key like schema_version=1 at the top level which can be used to check when the user should receive a warning about outdated rules and how to revise them.)

NobodyXu commented 2 years ago

I'm not sure about configuration file splitting though. I've learned to follow YAGNI the hard way and I haven't yet had a solid rationale that convinces me it's worth the added maintenance burden and risk of overlooking a way it could make the design more vulnerable.

Maybe we should wait for user feedback. If the configuration file is simple enough, then splitting it definitely isn't necessary.

...but I will want to think about whether it's worthwhile to let users extend the default config rather than completely overriding it, given that things like adding new profiles would require them to add new aliases, and making the rules more nuanced will probably result in a new schema version for the config file.

Agreed, we should also provide a sensible default configuration.

ssokolow commented 2 years ago

Agreed, we should also provide a sensible default configuration.

We should aspire to have 99% of users not needing anything but the defaults.

ssokolow commented 2 years ago

Just to get it noted down, the main thing I'll want to sit and think about once I've got a proto-v0.1 to experiment with is the schema for the config file.

With needing to support subcommands (eg. fetch) and flags (eg. --offline) as means to control whether things like network access should be granted, the draft design shouldn't be the final one.

To achieve that properly, I'll probably want to sit down and write out a sheet with all the different commands (eg. cargo, npm, etc.), subcommands, and flags that are relevant to the behaviour of the sandbox, and then brainstorm on the design which can describe them in the cleanest way.

...either that or just treat schema version 1 as temporary and accept that, for 99% of use-cases, it should be expressive enough, even if it's a little inelegant.

NobodyXu commented 2 years ago

Maybe we can have a section like “[profile.cargo-fetch]” and override and extend configurations provided in “[profile.cargo]”?

NobodyXu commented 2 years ago

I just realised today that Godbolt also uses sandbox internally.

After a quick look at the source code, I found the firejail configuration they use.

ssokolow commented 2 years ago

Maybe we can have a section like “[profile.cargo-fetch]” and override and extend configurations provided in “[profile.cargo]”?

I considered that briefly, but it feels like it'll get verbose and/or complicated quickly with the interactions of things like cargo defaults (no network), overridden by fetch and build and run (network), overridden by --offline (no network) if we don't sit down and carefully plan it out.

That's why I want to research all the commands I can think of supporting first... so I can sit down and carefully plan properly.

(Remember, my goal is a sandboxing thing that Just Works™ if you stick nodo onto the beginning of your usual commands, so it has to be expressive enough to handle cases like that.)

ssokolow commented 2 years ago

...also, maybe I should give it a -v option that causes it to print something like nodo: profile=cargo +net -gui -dbus root=/path/to/detected/root as the first line of output as part of a comprehensive plan to steer users away from just blindly dropping the nodo if a build fails in the hope that will make it work.

NobodyXu commented 2 years ago

Maybe we can have a section like “[profile.cargo-fetch]” and override and extend configurations provided in “[profile.cargo]”?

I considered that briefly, but it feels like it'll get verbose and/or complicated quickly with the interactions of things like cargo defaults (no network), overridden by fetch and build and run (network), overridden by --offline (no network) if we don't sit down and carefully plan it out.

That's why I want to research all the commands I can think of supporting first... so I can sit down and carefully plan properly.

(Remember, my goal is a sandboxing thing that Just Works™ if you stick nodo onto the beginning of your usual commands, so it has to be expressive enough to handle cases like that.)

One issue with that is that cargo can be extended…

ssokolow commented 2 years ago

Of course, but if you're extending Cargo, then it's perfectly reasonable for you to be expected to extend your sandbox definitions. That's one reason the syntax needs to be simple and easy to work with.

...still, I do intend to include ready-made definitions for some of the most common third-party subcommands.

NobodyXu commented 2 years ago

...still, I do intend to include ready-made definitions for some of the most common third-party subcommands.

Yeah.

That's why I want to research all the commands I can think of supporting first... so I can sit down and carefully plan properly.

I tried to list alll commands supported by cargo's stable channel and split them into 4 categories:

Commands that requires network access and modifies $HOME/.cargo:

    install              Install a Rust binary. Default location is $HOME/.cargo/bin
    login                Save an api token from the registry locally. If token is not specified, it will be read from stdin.
    logout               Remove an API token from the registry locally
    search               Search packages in crates.io
    uninstall            Remove a Rust binary

Commands that creates new project (need special handling since Cargo.toml does not exist):

    init                 Create a new cargo package in an existing directory
    new                  Create a new cargo package at <path>

Commands that only needs access to the project:

    verify-project       Check correctness of crate manifest
    version              Show version information
    check                Check a local package and all of its dependencies for errors
    clean                Remove artifacts that cargo has generated in the past
    clippy               Checks a package to catch common mistakes and improve your Rust code.
    d                    alias: doc
    doc                  Build a package's documentation
    fmt                  Formats all bin and lib files of the current crate using rustfmt.
    locate-project       Print a JSON representation of a Cargo.toml file's location
    metadata             Output the resolved dependencies of a package, the concrete used versions including overrides, in machine-readable format
    pkgid                Print a fully qualified package specification
    read-manifest        Print a JSON representation of a Cargo.toml manifest.
    rustdoc              Build a package's documentation, using specified custom flags.

Commands that requires network access, access to the project and $HOME/.cargo:

    b                    alias: build
    bench                Execute all benchmarks of a local package
    build                Compile a local package and all of its dependencies
    c                    alias: check
    fetch                Fetch dependencies of a package from the network
    fix                  Automatically fix lint warnings reported by rustc
    generate-lockfile    Generate the lockfile for a package
    package              Assemble the local package into a distributable tarball
    owner                Manage the owners of a crate on the registry
    publish              Upload a package to the registry
    r                    alias: run
    run                  Run a binary or example of the local package
    rustc                Compile a package, and pass extra options to the compiler
    t                    alias: test
    test                 Execute all unit and integration tests and build examples of a local package
    tree                 Display a tree visualization of a dependency graph
    yank                 Remove a pushed crate from the index
    update               Update dependencies as recorded in the local lock file
    vendor               Vendor all dependencies for a project locally

Since the commands above all build the project, they also might need to download the dependencies or build.rs might access the network.

Also, some users might use sccache (yeah, I use it) and it can be configured to distribute the compilation jobs and use cache stored in external machines.

NobodyXu commented 2 years ago

Here are some cargo extensions that I know:

Commands that requires network access, access to the project and $HOME/.cargo:

They are need to compile the project, so they would also need to download dependencies, which requires network.

ssokolow commented 2 years ago

Would you mind running through the commands that require network access to check how many follow the convention of having an --offline flag which the sandbox should recognize as a signal to not allow network?

NobodyXu commented 2 years ago

Would you mind running through the commands that require network access to check how many follow the convention of having an --offline flag which the sandbox should recognize as a signal to not allow network?

It seems that all the builtin commands of cargo that requires network access has --offline in their help page.

And then I checked cargo --help and found that --offline is one of the standard flags that every cargo subcommand has.

NobodyXu commented 2 years ago

For the cargo extensions I mentioned:

does not have --offline.

ssokolow commented 2 years ago

In that case, I suppose the question is whether we should offer a "Just Works, but less secure" mode of operation where any unrecognized subcommand gets network access (unless --offline is seen) and whatever solution we come up with for $HOME/.cargo.

(eg. I'm thinking that we'll want to force $HOME/.cargo/bin to read-only access for every subcommand that has $HOME/.cargo access except install and uninstall.)

NobodyXu commented 2 years ago

eg. I'm thinking that we'll want to force $HOME/.cargo/bin to read-only access for every subcommand that has $HOME/.cargo access except install and uninstall.

I will add that most commands except for cargo publish does not access $HOME/.cargo/credentials.

And we should also make $HOME/.cargo/config.toml read-only.

NobodyXu commented 2 years ago

In that case, I suppose the question is whether we should offer a "Just Works, but less secure" mode of operation where any unrecognized subcommand gets network access (unless --offline is seen) and whatever solution we come up with for $HOME/.cargo.

I think we can implement this in v0.1 MVP, then improve on it in the subsequent revisions.

ssokolow commented 2 years ago

Yeah.

Also, for a later v0.1.x, it'd probably be a good idea to do a survey of Cargo subcommands to identify what inside the project root could be forced to read-only access for which commands.

(eg. I don't imagine most commands will execute anything like build.rs that might need permission to modify the contents of src, and we can whitelist the things allowed to modify Cargo.toml.)

NobodyXu commented 2 years ago

Yeah.

Also, for a later v0.1.x, it'd probably be a good idea to do a survey of Cargo subcommands to identify what inside the project root could be forced to read-only access for which commands.

(eg. I don't imagine most commands will execute anything like build.rs that might need permission to modify the contents of src, and we can whitelist the things allowed to modify Cargo.toml.)

Totally agreed.

Modifying the project other than target should be forbidden.

Honestly, most of the build.rs don't modify the source. Binding generator like bindgen usually outputs the generated binding to target, and the file will be included in a predefined module in src/ using something like include_file (can't remember the name) include!(concat!(env!("OUT_DIR"), "/binding.rs"));

Though we also need to take into account that user can configure cargo to use a custom "target" using --target-dir.

NobodyXu commented 2 years ago

Another thing I want to add to the configuration is whitelist_path.

Some users like me who use sccache would like the cargo build to be able to access the cache.

ssokolow commented 2 years ago

That's why I'm not going for full whitelisting until at least 0.2.x. You wind up deep in the weeds reinventing parsing every command's arguments if you try for that.

Well down the curve of diminishing returns. For the most part, I'm just thinking about this stuff now because it will help to reduce how many different revisions the config schema will have to go through to keep up with demands.