Intermediate C SDK to simplify interop / FFI with other languages

[radu-matei]

Spin currently supports two language SDKs — Rust and Go. While the two SDKs have feature parity, the way they are implemented is very different:

• the Rust SDK is directly built on top of Rust bindings generated by https://github.com/bytecodealliance/wit-bindgen, with a thin layer of macros on top to simplify writing an entry point.
• Go does not have support in WIT bindgen, so the Go SDK uses the C bindings and CGO to translate between the low level types that represent request data and high level Go objects. Additionally, since there are multiple sets of WIT files in Spin, generating multiple sets of C files means managing them separately in how we build the SDK.

As we try to build more Spin SDKs for languages that don't currently have WIT bindgen support, we expect to see a few issues with individually generating the same C bindings for each project:

• duplicating the generation of C bindings from WIT files
• there are multiple sets of WIT files currently in Spin (HTTP trigger and outbound HTTP, Redis trigger and outbound Redis, configuration), and we have to generate bindings for them separately. Adding a new interface requires updating how we generate the bindings in each project that has C FFI support
• linking the canonical ABI implementation multiple times from multiple interfaces could result in linking errors

A lot of these issues would be either simplified or entirely avoided if we had an intermediate C SDK that aggregated all bindings, and had a common representation for the objects used (HTTP objects, for example) for languages that needed to interop with the C bindings.

Then, the Go and future SDKs would depend on this friendlier SDK rather than on the lower level auto-generated bindings.

Thoughts?

[Mossaka]

I certainly agree that building language SDKs out of the generated C bindings would be extremely hard to maintain.

In addition to the proposed intermediate C SDK idea, I want to offer some alternatives:

1. Contributing to wit-bindgen to add more language bindings is the obvious and ideal way, but it's cost to engineering efforts is no joke. I would be willing to contribute to wit-bindgen to support more guest langauges, but I understand this is unreal to acheive in the short term.
2. How about an intermediate Rust SDK? Here is a list of Rust interos. As you can see, many languages have packages to directly call into Rust functions. Of course they abstracted Rust C FFI but I think the team's expertise in Rust would expediate the SDK authoring proces, as we get all Rust features on building such a SDK. If there are no libraries for the wanted languages to call Rust function directly, you can always generate C bindings and then use them.

[squillace]

I also support your move, but I, too, would suggest we collaborate as a community to add go bindings to wit-bindgen. I get the problem, however.

[radu-matei]

Adding Go support to WIT bindgen would be amazing, and we're happy to help that effort! This is not trying to replace or make that effort obsolete.

Having an intermediate (C or Rust) SDK would help if we want to support other languages we that don't have yet support, plans, or bandwidth to support directly in WIT bindgen.

(I doubt we'll have for example Swift support in WIT bindgen, but it would be pretty cool to support it in Spin beyond a Wagi component.)

[squillace]

sounds great to me.

[elewis787]

The call out for standard objects would be helpful. Currently running into issues with Zig which does not have a fully implemented std http package. Http packages that I have been using ultimately rely on net.Address which uses Sock.Addr under the hood. When targeting WASI, the Sock.Addr functions are not implemented ( related to WASI socket support ) My current path is to build a light weight http library that does not leverage net.Addr to get around this ( i.e - just the http objects )

[Mossaka]

(I doubt we'll have for example Swift support in WIT bindgen, but it would be pretty cool to support it in Spin beyond a Wagi component.)

That is a good point! Do you have a list of languages you want to support in mind? I would suggest adding as much mainstream languages as possible: C#, Js (Ts), Python, Java, Swift, and C++