Bindgen package for easily calling functions with arbitrary parameters

[clarkmcc]

Is your feature request related to a problem? Please describe. Calling guest functions and passing non-number values is a bit of a pain as noted here (in issue https://github.com/tetratelabs/wazero/issues/258). It involves allocating guest memory from the host, then passing the pointers to the guest, the guest reads the data at those pointers, does some work, then allocates a response and the host then reads the response as raw bytes. I am proposing we introduce a bindgen package that simplifies the host -> guest interaction. Later we can look at introducing something to assist in guest -> host calls.

To start, I will be working on a PR that supports this for Rust-based guest WASM modules, and in the future, there may be a way we can provide the same integration for TinyGo guests.

Describe the solution you'd like Most of the heavy lifting (and thinking) has already been done in wasmedge-bindgen. This existing repository relies on two guest functions and two host functions to help pass arguments around properly:

• (Host) return_result - Allows the guest to return a pointer and length of the result data.
• (Host) return_error - Allows the guest to return a pointer and length of the error message.
• (Guest) allocate - Allows the host to allocate guest memory
• (Guest) deallocate - Allows the host to deallocate guest memory

I propose we add the host functions to a new "wazero-bindgen" module. The Rust library providing the bindgen macros will optionally compile the two guest functions (allocate and deallocate) if a feature is enabled.

It would just be a matter of:

• Creating a new Go package like this and providing a method on the Module interface so that we can call a function and pass in the arbitrary arguments.
• Creating a new Rust repository (perhaps tetratelabs/wazero-bindgen-rs) that contains the guest side of the bindgen like this.

Questions I plan on implementing this for a personal project, if this is of interest in the community, then it might make sense to get feedback on how the interfaces should look before I get started.

• How would we want to allow the host to call the guest functions with bindgen arguments? Perhaps we could add a method to the Function interface like CallBindgen.
• How could we provide the same support for TinyGo? Go would not support the macro approach that Rust would use, but maybe we do something like this

  //export add
  func add(ptr, size uint32) bindgen.Function {
  return bindgen.Wrap(ptr, size, func(a, b) int {
      return a+b
  })
  }

[clarkmcc]

Just added a proof of concept draft PR: https://github.com/tetratelabs/wazero/pull/627.

Rust

#[wazero_bindgen]
pub fn greet(name: String) -> Result<(), String> {
    Ok(format!("Hello {}", name))
}

Go

// Create the Wazero runtime
ctx := context.Background()
r := wazero.NewRuntime()
defer r.Close(ctx)

// Instantiate the bindgen module in the runtime before instantiating our WASM module.
// This will load in some host functions that are required.
bg, err := bindgen.Instantiate(ctx, r)
if err != nil {
    panic(err)
}

// Instantiate it in the runtime
module, err := r.InstantiateModuleFromBinary(ctx, wasm)
if err != nil {
    panic(err)
}

// Bind the bindgen instance to the module
bg.Bind(module)

results, err := bg.Execute(ctx, "greet", "Wazero")
if err != nil {
    panic(err)
}
fmt.Println(results[0].(string)) // Prints "Hello Wazero"

[clarkmcc]

We could consider implementing a Go host bindings generator for https://github.com/fiberplane/fp-bindgen. It would be more involved, but would probably be a better approach since there is an established spec based on msgpack which we could utilize from Go.

[mathetake]

sounds nice! I'm a bit concerned about the maintenance burden we would incur here, but yeah I think this seems valuable and helpful to the community.

@codefromthecrypt is OoO until tomorrow, so he will reply back in a day or so! Thanks!

[codefromthecrypt]

Hi, @clarkmcc thanks for taking time to work on this, and I can already see people interested.

TL;DR; I'm closing this as out-of-scope because something like this will be dissimilar from runtime code and this codebase is already very hard for us to maintain.

We also noticed some opportunities over time, including a tossed go helper #264. One thing we ran into on the go side was the tension between reflection and compilation, as well some investment needed on the codec side. Moreover there are specs in progress that are moving fast as well. When we do something, we need to know the types that will be available, how they behave when features needed, but not REC stage are turned off, etc. It will be a lot of work.

So, we have spoken about something like this, but can't do anything right now because all the above takes a lot of focus and we need it for the actual runtime. How the code moves which will be independent of the "vm" except possibly some hooks or otherwise. Due to our stepping back on this (regardless of need), it allows others to go first, like you have.

For your planning purposes, when(if) we do a binding library it would be in a different repository for sure, and start with Go, and consider a few other things. Definitely it won't be here and even that future repo may or may not be a mono-repo wrt binding languages.. yet another decision to make right?

If you open a repo on your own, I can watch and help as I can, but for the same reasons we aren't doing this now, I can't dedicate a lot of focus to this. However, similar to https://github.com/inkeliz/karmem I can help spread the word! Also, you may find others who are interested to help on your work, too, and who knows where it can land!

In any case, knowing this isn't going to be in this repo, we should close this as out-of-scope. I hope you understand.

[clarkmcc]

@codefromthecrypt that's understandable. Utilities like this affect the developer experience with the runtime so I think it does need to be addressed at some point in the future, but until then, it can certainly be worked around. Thanks for taking the time to give feedback!