Allow Wasm compiled plugins and builtins

[anderseknert]

The current plugin architecture requires both knowledge of Go, and for each plugin to be compiled together with OPA - essentially requiring a custom fork to be maintained. Needless to say, this poses a pretty high barrier to entry, and there's currently not a whole lot of public plugins to be found.

With the introduction of a Wasm runtime shipped with OPA, we should be able to use that for purposes outside of policy evaluation. If we allowed:

• Plugins written in any language compiled to Wasm.
• Custom built-ins written in any language compiled to Wasm.

We should be able to:

1. Increase the number of developers able to write plugins and custom functions for OPA.
2. Make plug-ins truly modular, decoupled from the OPA build process.

[srenatus]

So far, there has been no way to do networking from wasm modules. Looks like there's some recent efforts in experimenting with that in WasmEdge, see https://github.com/WasmEdge/WasmEdge and https://github.com/second-state/wasmedge_wasi_socket. Pretty experimental, though. No support for this in wasmtime yet, or perhaps even, soon.

[juntao]

Hi, I am the maintainer of WasmEdge. :)

The WasmEdge network socket API now supports Rust and JavaScript as front end languages. We are also adding DNS and TLS support.

https://github.com/second-state/wasmedge_wasi_socket/tree/main/examples

https://github.com/second-state/wasmedge-quickjs#http-request

Perhaps an place to start is for OPA to be able to ship multiple alternative Wasm runtimes? That is to enable the user to choose from Wasmtime / WasmEdge or others. That provides runtime diversity (so that bugs / security issues in one runtime can be worked around by switching to another), and also enables users to take advantage of different feature sets in different runtimes.

[srenatus]

@juntao Thanks for reaching out! As mentioned above, you folks are making some interesting progress there.

On shipping multiple wasm runtimes, how would you imagine this in practice? I'm all in favour of runtime diversity, but building+testing 8 instead of 4 binaries and images seems like it would be cumbersome... also, as a user, I don't need to know that there's a thing called "wasmtime" in OPA, let alone having to choose between "wasmtime" and "wasmedge" 😅

Perhaps code for the second runtime could be integrated in a different way... we could keep it "under wraps", use it for testing and exploring, and unveil it once we have some sort of incentive to switch: like being able to use wasm-based plugins/builtins.

[juntao]

I think we could define a standard interface for the host (OPA) to communicate with the Wasm runtime. Envoy did that for proxy-wasm which allows multiple Wasm VMs to be plugged into Envoy.

https://github.com/proxy-wasm

Then, the OPA binary distribution could have a default Wasm vm so that user never needs to worry about "Wasmtime" or "WasmEdge".

But for the users who do want to select their Wasm VMs, we could have a config option. The Wasm VMs could be bundled inside OPA binary install package, or the user could install a compatible Wasm vm independently and configure it.

What do you think? :)

[srenatus]

Yeah we'd need come up with something of a spec there, you're right. IIRC the interested parties in envoy had to do quite some work to get the spec into shape. Also, maintaining SDKs for that then is no mean feat either. (We'd definitely need community contributions here.)

What makes me wonder -- is our desire unique enough that we need out own spec? Maybe there's something out there that would fit well enough? What comes to mind is piggy-backing on waPC or perhaps Atmo's Runnable API.

Another approach to keep the effort lower would be to use some common interface definition and generate code based on that. The only option I am aware of here is witx (interface types "to-be"), and that seems in a pretty WIP-y state too. 🤔

[juntao]

I think it may be easier to create a working example on one Wasm VM (preferably WasmEdge ;) and then generalize an API spec for all Wasm VMs.

Specifically, I think we need to

1 Execute a wasm function from OPA. That is most likely done via the GO SDK of the Wasm runtime.

https://www.secondstate.io/articles/extend-golang-app-with-webassembly-rust/

2 Call OPA functions from the Wasm program. That is done by registering the OPA GO function as external references in the runtime.

https://github.com/second-state/WasmEdge-go-examples/tree/master/go_ExternRef

Then, we need to create a Rust or JS (or WITX) API for devs to write the Wasm functions.

If you can point us to a simple use case (eg what the Wasm function should do and return to OPA), and then point us to the place where OPA calls Wasm (#1 above), we can start there. 🤓

[srenatus]

Sorry for dropping the ball here.

1 Execute a wasm function from OPA. That is most likely done via the GO SDK of the Wasm runtime.

We're not there yet: having wasm-provided builtin function plugins is not work anyone had started yet. It would be a decent-sized project, too, so it's not the simplest thing to slap a "help-wanted", "good-first-issue" label on 😅

Let's keep this issue open for tracking status, anyways. The path you've outlined there is a way to go, for sure! Thanks for engaging.

[juntao]

Yes, it would require design decisions from the host (OPA) and is not trivial.

Appreciate your consideration. Perhaps we can revisit this issue later when there is a clear use case. 🤓

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[srenatus]

OK so here's a thought: let's use WASI!

A custom builtin would look like this:

1. it's a WASI command (not reactor)
2. it accepts its arguments via ARGV, as JSON
3. its output is JSON, taken from its stdout

We would not need any special SDKs, WASI is pretty standard by now.

But each wasm custom builtin module would need a JSON parser if it wants to have more difficult argument types than strings (which can be parsed with a less-than-full-json parser).

There would be no support for set arguments or outputs -- they cannot be encoded in JSON. Or we use the AST JSON format, which includes sets like {"foo"}.

[juntao]

Hmm, you mean to run the Wasm program in a separate process outside of OPA (as opposed to embedding into OPA). You still need to read / write the STDIN / STDOUT from OPA.

Is it easier than building integration with the GO SDK? If so, then yes. :)

[srenatus]

As far as the plugins and builtins topic is concerned, I was only thinking about custom builtins here.

OPA would evaluate these using wasmtime or wazero. It would call them with the args accordingly, and read and parse their stdout, to understand the output. I've been using stdin/stdout causally here -- not on the CLI, but "as if", whatever WASI does there, to emulate it.

In the end, we'd be able to test wasm builtins by updating them to https://webassembly.sh and running them there as if they were CLI programs; or by running them on an actual CLI using the wasmtime CLI.

Hope that makes sense? 😄

The big win would be that you'd be able to add custom builtins to OPA without having to compile your own OPA binary, indeed.

[anderseknert]

This is really interesting 😃 A couple of questions:

1. Would it be possible to provide a "Wasi SDK" or library that included, JSON parsing, and maybe some other helpers? Thinking more of this now I guess it would be hard to jack into something like that from a programming language that later would be compiled to Wasm. Maybe it wouldn't matter much, as any language or framework ships with a JSON parser, or provides one via dependencies.

2. Since you mentioned you only considered custom builtin functions here, is there something making it difficult to leverage the same capability to deploy e.g. a decision logger plugin? Is it the aforementioned problems with networking, sockets... etc?

3. I don't think we can get away with not suporting sets. It's just such a common data type in Rego, that we'll need at least some way to have them included, even if just something simple like adding some identifier/symbols to the keys when serialized as JSON.

4. Is there any good "hello world"-like Wasi tutorial you could recommend? 🙂

[srenatus]

A few notes, I'll get back to this with more later:

1. Would it be possible to provide a "Wasi SDK" or library that included, JSON parsing, and maybe some other helpers?

I think many use cases wouldn't need a full-blown JSON parser. Only builtins that operate on objects and arrays would probably need it...? JSON strings are pretty much just strings, I suppose 😅

Also almost any language comes with a json library these days, I think. Of course, we should get a handful of examples together beforehand, create a WASI-based builtin with (Tiny)Go, one with AssemblyScript, one Rust... to ensure we're building something that fits the requirements.

4. Is there any good "hello world"-like Wasi tutorial you could recommend? 🙂

https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/ is the standard starter, I think. From there on, it depends on the language you're interested in. wasmtime has a tutorial, too: https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-tutorial.md (C/Rust).

[srenatus]

3. I don't think we can get away with not suporting sets. It's just such a common data type in Rego, that we'll need at least some way to have them included, even if just something simple like adding some identifier/symbols to the keys when serialized as JSON.

Yeah we could encode them as {"a", "b"}, and put the burden on the person who's writing a wasm custom builtin. Or we could turn them into arrays ["a", "b"] when passed to the wasm custom builtin and allow the builtin author to just use any off-the-shelf JSON lib for their language.

In the interest of simplicity, I'd go with the later.

[srenatus]

4. Since you mentioned you only considered custom builtin functions here, is there something making it difficult to leverage the same capability to deploy e.g. a decision logger plugin? Is it the aforementioned problems with networking, sockets... etc?

Not really, it's more that I hadn't thought about this before. But if can come up with a generic enough mapping to WASI then that would be an option, too.

[anderseknert]

Custom functions would be a great start, so I'm fine with keeping the initial scope down to that. As for passing sets as arrays, I guess that's fine. The Rego compiler/runtime would still complain if a set was expected but not provided, and I guess we could do something similar to ensure type coersion on the return value, if a custom builtin is documented as returning a set.

[srenatus]

That's a good point, btw, custom builtins provided via wasm modules need some kind of type annotation.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.