Consider using protobufs or cap’n proto to cross language boundaries

[mistermoe]

I wonder if using protos would streamline passing complex structures over language boundaries.

1. Write .protos. I suppose 2 for each function. One for args. And one for return value
2. Use protobuf tooling to autogenerate concrete types / classes in rust & desired target lang
3. send protos back and forth

This way there’s also a single source of truth.

@frankhinek suggested exploring cap'n proto which could be a good option as well

[michaelneale]

cc @decentralgabe I think you (was it you?) was mentioning something about this?

[decentralgabe]

@andresuribe87 suggested json schema, protos, or open api

[phoebe-lew]

@mistermoe For

3. send protos back and forth

I think this is a separate discussion, which we should have as well, but can be decoupled.

There's

1. What do we use for data definitions (json schema, protos, open api, capnproto (? not familiar with this one), ...)
2. how do clients communicate with the server/PFIs (RESTful, JSON-RPC, gRPC, ...)

I assume the original comment was referring 1., data definitions?

[phoebe-lew]

Also note capnproto's supported languages in our eval: https://capnproto.org/otherlang.html

[andresuribe87]

For 1, while I'm tempted with capn proto, I'm concerned with the maturity, tooling, and community around it. I think we can actually switch out between different serialization frameworks easily later on, so I would go for best devX. To me, that's proto (most likely because I've worked with it so much).

For 2 I would actually recommend zero mq.

[mistermoe]

I was suggesting exploring protos as the serialization / deserialization mechanism for arguments that are non-primitive data structures when calling a rust function from a different language, within a binding.

Contrary to a network boundary

Will provide an example in the morning!

[mistermoe]

Hello 1 week later. Contrived example. Imagine we have something like this written in rust:

/// Represents a Decentralized Identifier Key (DidKey).
pub struct DidKey {
    // Define the properties of DidKey here
}

/// Represents the options for generating a `DidKey`.
pub struct GenerateOptions {
    /// The algorithm to be used in the generation. 
    /// This is a required property.
    pub algorithm: String,

    /// The curve to be used in the generation.
    /// This is an optional property.
    pub curve: Option<String>,
}

impl DidKey {
    /// Generates a new `DidKey` based on the provided options.
    /// 
    /// # Arguments
    /// 
    /// * `options` - An instance of `GenerateOptions` containing:
    ///     * `algorithm` - A string specifying the algorithm to be used.
    ///     * `curve` - An optional string specifying the curve to be used.
    ///
    /// # Example
    /// 
    /// ```
    /// let options = GenerateOptions {
    ///     algorithm: "example_algorithm".to_string(),
    ///     curve: Some("example_curve".to_string()),
    /// };
    /// let did_key = DidKey::generate(options);
    /// ```
    pub fn generate(options: GenerateOptions) -> Result<DidKey, &'static str> {
        // Implementation of the generate function here.

        // Check the validity of provided options, create the DidKey, etc.
        // If successful, return Ok(DidKey), otherwise return an Err with a message.

        Ok(DidKey {
            // Initialize the DidKey properties here
        })
    }
}

fn main() {
    // Example usage of the DidKey::generate function

    let options = GenerateOptions {
        algorithm: "example_algorithm".to_string(),
        curve: Some("example_curve".to_string()),
    };
    match DidKey::generate(options) {
        Ok(did_key) => {
            // Handle the successfully generated did_key here
        }
        Err(err) => {
            // Handle the error message here
            println!("Error: {}", err);
        }
    }
}

let's say we want to create JS, JVM, and Kotlin bindings for ^.

Honing in on JS specifically, we'll have a binding on the Rust side of the fence and a binding on the JS side of the fence. I could be making this harder than it really is but, I'm not entirely sure how we'd pass a non-primitive arg type from one side of the fence to the other. Ideally there would be some declarative way to define the API surface or at least the arguments each function takes and the expected return value that could then be used to autogenerate the necessary types on the target language side so that we don't have to manually handroll that stuff

here's an example of how a JS binding would work with primitive args using wasm_bindgen:

Rust:

use wasm_bindgen::prelude::*;

#[wasm_bindgen]
impl DidKey {
    #[wasm_bindgen(constructor)]
    pub fn generate_js(algorithm: String, curve: Option<String>) -> Result<DidKey, JsValue> {
        let options = GenerateOptions { algorithm, curve };
        DidKey::generate(options).map_err(|e| e.into())
    }
}

JS:

import { DidKey } from "./autogenerated.js";

try {
    const didKey = DidKey.generate_js("example_algorithm", "example_curve");
    // Now didKey is an instance of DidKey created by Rust, but usable in JS!
} catch (error) {
    console.error("Error generating DidKey:", error);
}

[amika-sq]

wasm_bindgen looks like it supports exporting Rust types: https://rustwasm.github.io/wasm-bindgen/reference/types/exported-rust-types.html

Haven't had an opportunity to play around with it yet, but it should remove the need for a separate generate_js interface to bridge the two worlds together.