Go lang support - current and future

[happybeing]

@b-zee Sorry to bother you but wonder if you have a moment to comment.

I'm looking at extending rclone (written in Go) to use Autonomi. I hope to write most of this in Rust and call it from Go so wondering if you have any comments on support for calling the Autonomi APIs from Go.

So far there seem to be two options:

• use CGo (C API support for Go) and write a Go package to wrap my Rust library
• compile my Rust library (which calls the Autonomi Rust API) to WASM and look at calling that from Go (e.g. using wasmer-go)

[b-zee]

I'm not sure if the wasmer-go runtime is capable of doing the same things we can do from the browser (websockets). If that is the case, then yes, wasm might be a possibility. But I think in terms of maturity and performance, using FFI is for sure the better route.

In the future we might want to look into WASI, but that is kind of tricky from a browser perspective. In theory the browser could run WASI and allow sockets to open etc., and then the exact same WASI would run from the terminal with something like wasmer etc. But that's a route we haven't fully explored yet.

[happybeing]

Thanks @b-zee. To clarify, your WASM target relies on WebSockets in order to do network operations?

So the Autonomi libs won't work unless WebSockets are supported in wasmer-go?

[b-zee]

We rely on libp2p-websocket-websys. Which relies on the availability of Web APIs (via web_sys). In other words, it calls into the browser JS runtime to create websockets.

I'm not entirely sure what wasmer-go supports with regards to these Web APIs (or even JS APIs (see js_sys crate)).

[b-zee]

I think the wasmer-go runtime does not support anything out of the box, so you're probably out of luck there.

[happybeing]

Yep, looks like WASM isn't a good route for Autonomi outside the browser then.

Thanks for your help, very useful as always.

I guess this means that for most languages it's back to creating Autonomi ffi bindings per language.

One last question (I hope!)... does this mean that Autonomi JS support only works in the browser? I ask because of course in the past it was able to be used from NodeJS, so a backwards step for some reason.f

The author of rclone pointed out https://pkg.go.dev/github.com/tetratelabs/wazero is something he's been looking into using to allow WASM backends to be loaded (e.g. as plugins). I don't know if it makes any difference to my question, but CGo is not ok as they don't want to include it so I'm running out of options.

[b-zee]

For now, our JS API will only work from within the browser. This is something I actually overlooked up until a few weeks ago. We aim for a browser package, because it's super magical to connect to Autonomi from a website. However, NodeJS suffers the same problem as some WASM runtimes will: they won't all support the same Web APIs. There are some ways to shim WebSocket functionality to make libp2p work, but I haven't actually gone that route yet.

FFI is the only solid/mature option here. If they don't support it, you're out of luck. Are all their backends native Go implementations? Thinking about it, they might mostly be Go wrappers around REST APIs.

In theory there is the option to write a Autonomi Go client, using libp2p to connect to a peer and retrieving data. But that's going to be pretty involved I'm assuming.

[happybeing]

Thanks @b-zee I appreciate you taking the time when things must be crazy hectic - are they ever not?! :rofl:

You are correct that most rclone backends are RESTful web, each supported using customised Go in rclone/backend. They have some neat extras such as local, VFS and you can also mount your backend as a FUSE drive, and lots of other projects use rclone as an API (there's even an Android app). So it is a very attractive thing to support but there are a couple of other hurdles before ffi is an option so I'm still not sure if I can start work on this.

Thanks again. Hope all is well with you.