Support Host Functions

[bhelx]

We want the JS-PDK to support host functions. They should appear as imports in the JS world. We could come up with some kind of convention like:

import {host_func1, host_func2} from "extism:host";

But the host won't be able to provide them unless they make it into the wasm binary in the imports table. We could try a similar trick to exports. What it does is at compile time, it parses the javascript and looks for the exports, then it generates some "shim" wasm instructions that eval the function by name: https://github.com/extism/js-pdk/blob/main/crates/cli/src/main.rs#L115

Then it adds the exports to the exports table pointing at the shims: https://github.com/extism/js-pdk/blob/main/crates/cli/src/main.rs#L115

We could try something similar with imports. What might make it tricky is, i think, the function indices in the module start with imports, so you can't really append to them without re-writing every function index.

At this point, we're basically writing and maintaining our own linker. So perhaps we should look into a way we can use a tool to statically link the wasm modules at compile time and handle all this merging and re-indexing of functions.

This work may collide with https://github.com/extism/js-pdk/issues/19 if we decide to go down that path.

[bugzpodder]

Since we have a Host global, would it make sense to use a similar approach instead of "extism:host"? Like HostCall.host_func1 or HostCall.host_func2? I also peeked at other PDKs, they are generally light on details on how host function calls are implemented and used other than the default inputString().

If I am a plugin developer, and I want to to call some host_func1 it would be great if I can copy+paste/import some kind of definition so I know for sure what I am calling. But I haven't played with plugins so not sure if that's reasonable suggestion or not.

For exports I see that your code reference looks for a module.exports by eval some JS which seems fairly reasonable to me. Also see https://github.com/dylibso/pg_extism/blob/main/plugins/chatgpt/script.js#L1 which may or may not be useful.

[bhelx]

Yes, that's a good point. I think putting the imports on the Host object or having another global makes sense to continue with. So I'd be open to any of those options.

The tricky issue will still be getting the imports defined into the wasm file. So we'll still need to parse something ahead of time and do a similar trick we do with exports. We could also have some conventional structure you're expected to define as a global in the top of your module. Kind of like how rust works with externs:

#[host_fn]
extern "ExtismHost" {
    fn hello_world(input: String) -> Json<Output>;
}

// ^ that gets expanded to this:
extern "C" {
    fn hello_world(input: i64) -> i64;
}

The compiler will need to know what all the imports are that the plugin needs and what the wasm type signature of each host function is. So it will need to parse the JS (or wherever you want to put these declarations), look for these imports and signatures, then compile in the imports into the wasm binary.

[bhelx]

Started work on this and we have a new proposal https://github.com/extism/proposals/pull/16

[bhelx]

I believe we can call this done