FR: Rust Result impedance matching

[crusso]

The Rust and Motoko Result are isomorphic but don't match, leading to awkward interop with Rust canistes (i.e. all our system canisters).

What should we do about it?

1. Introduce a new Result library and deprecate, but retain the old one? There are few dependencies on Result in Base (List/Array/Buffer) and we could actually add support for the new and retain support for the old lib.

2. Exploit the fact that the Candid to Motoko binding is many to one, and import Rust Result as Motoko Result. This only solves half the problem as the Motoko to Candid translation is one to one, so Motoko Results would still be seen as incompatible from Rust.

3. Remap labels #err and and #res to #Err and #Res on parsing. Motoko would start magically working with Rust canisters, but stop working with older Motoko canisters. Relies on whole program compilation (which we do).

4. Indicate canister source (Rust/Motoko) on import and selectively remap.

5. @ggreif idea of extending Candid with synonyms for variant tags, record labels. Not sure how that works.

6. Add an (optional) conversion clause to the public actor method. The body would see the converted variant, and the external world the unconverted one.

Any other suggestions?

[nomeata]

Change Candid to compare tags name case-insensitively? (That's a seriously breaking change, though.) It's similar to 3 in that way.

[ggreif]

Expanding on item 5) above... My suggestion is to allow alternate spellings on variant tags and field labels:

• variant { err|Err|ERR|Failure : <type> }
• record { field|Field : <type> } with the frontmost spelling being the canonical one and being inserted into messages, while the other spellings being accepted when decoding the message.

[rossberg]

I don't think it makes sense trying to solve this on the Candid level. It is mostly coincidence that Rust and Motoko are even that close on this. Plenty of other languages made different choices, e.g., I have seen Success | Failure, and other schemes. Ultimately, downstream apps have to adapt to whatever choices were made for a given interface.

The only way to reasonably unify interface styles is by having Candid-level style guides. But inevitably those will not happen to match most languages.

[chenyan-dfinity]

I would vote for 4, so that we don't do any magic at the Candid level.

import A "rust_canister:abc";

We can do similar things for Rust CDK.

[rossberg]

@chenyan-dfinity, we introduced Candid to abstract away from language specifics. This would be reintroducing them. No such approach can possibly scale. When designing any language-related mechanism for the IC, we shouldn't just account for the situation today, but consider the intended future where 10s or 100s of languages flourish on the IC. Any magic import-site mechanism would ultimately require every language to know about every other -- clearly infeasible. Or if they don't, then certain languages end up privileged in the ecosystem, which is highly undesirable.

Really, we must not allow one language's choices to leak through into other languages' semantics. That'd be a total dead end. The only way to solve this is by introducing language-agnostic Candid-level conventions. And then every language can implement transformations for these conventions. That reduces interop from a global O(n^2) problem to a local O(n) problem, which was the point of Candid.

[chenyan-dfinity]

we introduced Candid to abstract away from language specifics. This would be reintroducing them.

It kind of happened already. In Motoko, we can import actor class from another Motoko file without going through Candid.

The only way to solve this is by introducing language-agnostic Candid-level conventions.

I doubt this convention can ever exist. How can a language-agnostic convention cover all/most language features.

To avoid the O(n^2) problem, option 2 seems reasonable: Each host language specifies conventions on how they map Candid type to their host type, possibly with user intervention, when there are one-to-many mappings from Candid to the host language. We can extend the IDL-Motoko spec to cover the Rust result case.

For Rust CDK, we can have similar conventions. More user intervention is expected there. For example vec record { int; text } in Candid can map to Vec<(Int, String)>, HashMap<Int, &str> or even &[(Box<Int>, Arc<String>)] in Rust. We need users to specify how we want to map these types.

[rossberg]

It kind of happened already. In Motoko, we can import actor class from another Motoko file without going through Candid.

That seems unrelated. Imports are a Motoko mechanism that has nothing to do with Candid as such. Importing a module does not involve Candid either.

How can a language-agnostic convention cover all/most language features.

Right, it can't. Hence Candid has a limited, canonical set of types that languages have to translate from and into. They cannot expect that every of their features can be mapped to Candid, because not every other language can understand every of their features. Candid is the l.c.d. of sorts.

But you're not magically solving the problem by trying to push it to another layer. If you cannot solve it with Candid, then the likelihood is that this problem is unsolvable in principle.

To avoid the O(n^2) problem, option 2 seems reasonable: Each host language specifies conventions on how they map Candid type to their host type, possibly with user intervention, when there are one-to-many mappings from Candid to the host language. We can extend the IDL-Motoko spec to cover the Rust result case.

As long as the Motoko spec or implementation needs to know anything about Rust conventions, you are still proposing a non-modular O(n^2) approach.

For Rust CDK, we can have similar conventions. More user intervention is expected there.

Yes, it's totally fine to enable users to specify app-specific mappings for Candid types. That is something they can do on a case-by-case basis, with domain-specific knowledge about the specific service they are interoperating with. That's very different from expecting generic knowledge about "all the services" from the language itself.

[chenyan-dfinity]

As long as the Motoko spec or implementation needs to know anything about Rust conventions, you are still proposing a non-modular O(n^2) approach.

The spec is only about how to map Candid type to the host language. In fact, it's hard to know if a service is a Rust canister by just looking at the .did file.

For Motoko, we can say: all variant tags imported to Motoko will be lowercased, as that's the convention in Motoko. When serializing the message, we use the original tag in Candid. The downside is that it's a breaking change.

Or we can do nothing here, and setup a config language to allow users to specify app-specific mappings from Candid to the host language.

The need for such config language has come up in many use cases: 1) fine tuning language bindings as we discussed here; 2) pretty printing, for example, how to display a blob type in different applications; 3) Candid UI, for a blob type, uploading a file vs inputting a hex string; 4) random value generator

[rossberg]

The spec is only about how to map Candid type to the host language. In fact, it's hard to know if a service is a Rust canister by just looking at the .did file.

Right, but the crucial question is: does the spec involve meta-level knowledge of Rust conventions? If yes, you have an N^2 problem.

Lower-casing all variant tags is somewhat borderline. The idea is ultimately based on specific knowledge of how Rust and Motoko conventions relate, specifically in the case of the result type. As a general thing, it won't help.

So I think a programmable Candid mapping (as a mechanism in Motoko, not in Candid!) is the right way forward.

[ggreif]

As a fact point, the result type e.g. in Haskell is type Either a b = Left a | Right b. I don't see a name transformation that does Err <-> Left, resp. Ok <-> Right.

[nomeata]

I'm sure we can find a hash function so that it has few dozen convenient hash collisions, if we can find an exhaustive list.