Foreign linking / FFI

[rossberg]

This is a tracking issue for adding the ability to Motoko to interoperate with foreign code on an intra-canister level.

Some high-level design considerations:

• Cross language interop would naturally be provided at Wasm module boundaries, by supporting cross-language linking.
• For that we'll probably need to define some language-agnostic ABI/IDL, similar to Candid but for modules.
• We should keep track of the ongoing interface types proposal for Wasm, which is trying to solve the same problem for the Wasm ecosystem at large. However, it's still in early stages, and not entirely clear how well it will suite our needs.

Implementation considerations:

• A prerequisite is that Motoko supports separate compilation; currently, the compiler performs whole-program compilation.
• The Motoko compiler already has some specialised linking mechanism for the language's runtime system (RTS), which is written in C. However, this is not exposed and probably not general or robust enough in its current form.
• As we're adding more features to the Motoko base library, more flexible linking against foreign modules may become relevant for the language impl itself, e.g., to incorporate existing math, unicode, or database libraries.

Related: #147, #452.

[nomeata]

Thanks for starting this!

Some initial high-level questions about strategy and requirements:

• How viable is it to build a solution that has not full support in the Wasm community? Would it make sense for something to emerge there (interface types? something better?), or alternatively develop and advocate for something completely in that space, instead of here?

• Do we aim for zero-copy interop (shared memory, fewer safety guarantees) or shared nothing (copying values in an out, better safety, lower performance)? Will developer be happy with something that doesn’t look like C-like linking?

• Do we want to use the same linking mechanism between motoko modules? You hint at it by saying that separate compilation is a prerequisite, but I am actally not sure that’s true – cross-language linking is more a package-level concept, isn't it?

[rossberg]

• How viable is it to build a solution that has not full support in the Wasm community?

Good question. Good integration with the larger ecosystem is very desirable, so if possible, we should try to stick as closely to the interface types proposal as possible. But some of the directions they are taking are a bit worrisome, I don't think we want to depend on the whole enchilada, and it's still a moving target. Also, as with the canister IDL, we may need some Dfn-specific features (e.g. Dfn types).

So this is in part a question of timing and in part one of goals/scope, neither of which is fully set or stable for the interface types proposal.

• Do we aim for zero-copy interop (shared memory, fewer safety guarantees) or shared nothing (copying values in an out, better safety, lower performance)?

I think we want something that is flexible enough to cover both scenarios.

• Do we want to use the same linking mechanism between motoko modules?

Ideally, yes, because that would more easily allow the symmetric inverse: linking Motoko modules from other languages. That is, the linker should be generic, as with regular OSes. It would be bad if Motoko set a precedent of creating its own one-way street. Of course, you could introduce another layer of "package" to mitigate, but that's more tooling complexity.

In general, the ABI should be flexible enough to allow encoding everything Motoko needs, even if the compiler may map various high-level language concepts in ways that are not directly interpretable by other languages. In some cases, it may also boil down to a calling convention distinction, perhaps similar to extern "C" in C++. With that, a module could be "hybrid", i.e., provide "extern" exports as well as Motoko-specific ones.

Clearly a lot of specifics to work out here.

[nomeata]

linking Motoko modules from other languages.

How would that work with GC? Would other languages call motoko_rts_init(), maybe providing hooks to integrate into their own heap management? Or would this only work in the share-nothing-separate-memory world? In that case, that seems unlikely to be the same mechanism for linking Motoko modules (which surely should share a memory).

To me, language interop linking (separate modules/memories for memory safety desireable, common denominator type system) seems so fundamentally different from Motoko compilation unit linking (same memory linking, tight coupling with compiler version by sharing memory structures, shared RTS) that I think we’d do ourselves a service to solve them seprately.

[rossberg]

Yeah, I doubt that you often want to share the Motoko heap with non-Motoko modules. But Motoko modules might need to share other memories with third parties, even if it's just scratch space. I doubt that shared-nothing covers all use cases.

I agree that we can probably solve these problems separately, i.e., one after the other. But ultimately it would be undesirable to keep Motoko in its own isolated garden with a duplicated interfacing and tooling infrastructure.

FWIW, I gather that the module-level ABI would be sufficiently low-level that there is no hurdle in mapping Motoko onto it. I think the main role will be defining conventions around sharing memory. (I am not a big believer of the hyper-complicated pie-in-the-sky universal conversion operator algebra idea that the interface types proposal got itself into. Rather, I'd expect any non-trivial data to be passed through / serialised into some shared memory.)

[crusso]

I wonder if its worth taking a look at http://people.cs.uchicago.edu/~blume/papers/nlffi-entcs.pdf

[crusso]

… and here https://www.cl.cam.ac.uk/~jdy22/papers/a-modular-foreign-function-interface.pdf

[rossberg]

@crusso, my understanding is that both of these are specific to interfacing with C, and how to embed C's type system into the host language.

[crusso]

@rossberg they are but that doesn't mean the ideas wouldn't transfer to whatever interface wasm provides. Plus, just interfacing with C or better, Rust, would already be huge win.

[rossberg]

@crusso, while there typically is a standard C ABI on OS level, no such thing exists in Wasm. And there are more fundamental things you need to worry about, esp who owns the memory (memories?), how is it shared, and where is it managed.

[nomeata]

What about a “low level raw WASM FFI”, that allows linking against, well, any kind of Wasm module? Would that be feasible?

[rossberg]

You mean just bare Wasm? Yes, that's sort of what I have in mind as a starting point. But then you can only pass scalar data.

[nomeata]

Hmm, the FFI could have a way of creating a shared memory, exposed to the Wasm module as an importable memory, and to Motoko as a mutable resizable buffer (with an abstract interface) Or, similarly, if the Wasm module exports a memory, this could be exposed to Motoko as a mutable resizable buffer.

Not particularly convenient, but preserves motoko type safety, and it would allow people to write a Motoko library that wraps such a wasm module (say sqlite) with a nice Motoko interface.

[lastmjs]

It's been a while since this issue has had any activity, what's the latest status of FFI for Motoko?

[rossberg]

Nothing to report. It isn't realistic to design anything without having a suitable basis agreed upon by the wider Wasm ecosystem to build on. Otherwise there'd be little hope that e.g. Rust or C would support it. That is the idea of Wasm interface types, which I hope will get us there, but they're still not done.

[lastmjs]

Okay, so basically waiting on Wasm interface types, I can track there, thank you

[ggreif]

Just a random thought, would wasm { ... } blocks fill in any use-case (other than providing an unsafe exit hatch)? If one can extrapolate from ancient C times, such a lowest-level construct gave many game programmers the extra bit of performance they needed to make tight loops feasible. (I am not proposing such a feature, and I can totally see how it would open a huge can of worms.)

[rossberg]

Shuddering at the idea aside, the main use case here is linking with libraries compiled from other languages, where inline assembly wouldn't help.