Migrate Oak Runtime to Rust

[tiziano88]

Consider moving (parts of) the Oak Runtime from C++ to Rust. Things like gRPC and Asylo-specific stuff would remain in C++, but the Oak Messaging and Node management (including WebAssembly interpreter) could be moved to Rust.

Blocked by #313

[tiziano88]

Assigning to @blaxill to investigate what else is needed here.

The goal is to be able to support the channel / messaging parts of the Oak Runtime, and also being able to write implementations of (pseudo-)nodes in Rust.

Currently we have support for no_std plus alloc, which brings in Vec, Box and similar (see https://doc.rust-lang.org/alloc/index.html).

We probably still need some synchronization primitives (mutex at least), and threads, which I expect will be the hardest.

cc @daviddrysdale @annasapek

[tiziano88]

Also things like HTTP calls may have to call back to C++, which would be quite annoying. But maybe there's a way of linking Rust std http client calls to methods in the newlib implementation offered by Asylo.

[blaxill]

I agree no_std seems the most painless way forward, and I'm experimenting with simply wrapping Asylo threading primitives. Other alternatives that I'm not currently investigating:

• Use the sgx rust std lib, which relies on Fortanix's sgx libraries and so would require depending on Fortanix's sgx primitives.
• Maintain a separate rust std library for our targets. This would be a larger maintenance burden and is not currently needed.

[tiziano88]

FYI https://github.com/google/asylo/pull/46 but still under review. My understanding is that it does not really rely on the Asylo runtime / stdlib at all (which may or may not be fine for us).

The main point of using Asylo for us is to have the flexibility of targeting different enclave back-ends if and when Asylo supports it, while AFAICT Fortanix EDB is specific to SGX.

On the other hand, maybe we should define our own narrow interface in Rust of what we actually need from the underlying runtime (threads, synchronisation), and implement that ourselves for the various enclave back-ends.

re: wrapping Asylo threading primitives, isn't that basically the beginning of creating our own std library? Note that we don't need to implement every single API from std, just the ones we care about for our own code. As an aside, that's what https://github.com/apache/incubator-mesatee-sgx tries to do.

[blaxill]

• google/asylo#46 - using the rust_enclave target provided by this PR would allow rust enclaves that don't call back into asylo, and instead use fortanix sgx routines. This does however use fortanix's rust std library, which is sgx specific and may have different semantics for primitives (e.g. it's unclear to me at the moment how asylo/fortanix threads would interact)

• incubator-mesatee-sgx : similar to fortanix, provides a rust std lib for sgx.

Both of those methods are valid but are sgx specific- My understanding is ideally we want a rust std-lib that targets only asylo, so that we can target sgx but are not tied to it.

On the other hand, maybe we should define our own narrow interface in Rust of what we actually need from the underlying runtime (threads, synchronisation), and implement that ourselves for the various enclave back-ends.

Yes I think this would be useful as the underlying primitives may vary between hardware implementations. Although perhaps less important if we explicitly tie ourselves to asylo semantics.

re: wrapping Asylo threading primitives, isn't that basically the beginning of creating our own std library? Note that we don't need to implement every single API from std, just the ones we care about for our own code.

Yeah I agree, I just mean simple import wrappers to the asylo library for now.

[blaxill]

Unfortunately as Asylo doesn't export a C interface it makes it difficult to FFI call it directly. Fortunately Asylo provides a POSIX compatible interface: my strategy so far has been to create newlib pthread bindings (Asylo reuses these headers) with rust-bindgen, and then port over std::thread to use core rather than std.

The benefit of this route is

• we continue to only rely on Asylo,
• the Asylo POSIX interface (continues to) correctly handle the threading/mutex semantics we need. (E.g. providing basic cooperative threading via yielding on blocking mutex locks)
• if needed in the future, there is a clearer route to implementing rust-std for asylo (but it would be a lot of work)

[tiziano88]

If we were to take rust stdlib and fork it, would things line up correctly? e.g., if we were to take std::thread as it is and compile it for standard x86 linux, I expect it would call into libc at some point. Would this be the same signature and semantics of what the Asylo POSIX API expects?

When I tried compiling and linking all Rust stdlib against Asylo, things failed because a number symbols were not defined (see https://github.com/project-oak/oak/issues/322#issuecomment-555667321 for details). Asylo could implement the remaining symbols, or we may add stubs / mocks for those symbols ourselves in Oak that always trap (raise exceptions); as long as we use the parts of the stdlib that do not actually rely on those symbols, things should work fine (and this may or may not be the case for threads / sync), and we would notice (at runtime) if those functions do try to use the undefined symbols, at which point we can then decide whether to implement them properly or take another route.

[blaxill]

If we were to take rust stdlib and fork it, would things line up correctly?

We could fork stdlib and remove unsupported features. Perhaps we could merge the unix stdlib (for posix threads etc), and wasm stdlib (which provides empty stub instances for most types).

Asylo could implement the remaining symbols, or we may add stubs / mocks for those symbols ourselves in Oak that always trap

I think if we forked stdlib this shouldn't be a problem. Mocking symbols with traps would work but seems a longer route.

[tiziano88]

Should we write most of the runtime in Rust with no_std and just link to some high level functions that we implement in C? Basically define an interface of oak-runtime-specific functions that we cannot implement in Rust, and then we implement them in C / C++, currently using Asylo, but alternatively using other mechanisms (e.g. AMD SEV, or just plain no-enclave code).

e.g. malloc, free would map directly to the corresponding POSIX symbols, and similarly for the thread / sync primitives, but then we would also have higher level functions for HTTP / gRPC connectivity (mostly clients).

[blaxill]

Yeah I agree- Writing the rust runtime in no_std and calling to c for specific functions seems to be the most painless way forward.

My experience from last week suggests that for higher level functionality and in general external crates we would need to maintain a full branch of stdlib (due to POSIX expectations of rusts stdlib). If we continue to move more parts of the runtime into rust, we can revisit this later.

[blaxill]

Closing in favour of #540