Open tokahuke opened 5 years ago
I am working with capnproto-rust
for a pretty large project, and I also found using capnproto-rust
directly as it is somewhat difficult. I find that I always have to define the data at least three times:
I realize that working like this introduces performance penalty (One of the major selling points of capnp is having no encoding/decoding step). However, If I want to enjoy the no-encoding feature I will have to surrender to capnp and let it pollute my whole codebase with lifetimes and .reborrow()
-s. Sprinkle some async Futures into the mix and you get a real party.
I am willing to give up on performance in favour of clear and easy to maintain code. (At the same time I realize that other people might not want to).
To be more specific about what my problem is, I want to introduce an example from my code base:
struct FriendsRoute {
publicKeys @0: List(PublicKey);
# A list of public keys
}
2. Rust structure
```rust
pub struct FriendsRoute {
pub public_keys: Vec<PublicKey>,
}
pub fn ser_friends_route(
friends_route: &FriendsRoute,
friends_route_builder: &mut funder_capnp::friends_route::Builder,
) {
let public_keys_len = usize_to_u32(friends_route.public_keys.len()).unwrap();
let mut public_keys_builder = friends_route_builder
.reborrow()
.init_public_keys(public_keys_len);
for (index, public_key) in friends_route.public_keys.iter().enumerate() {
let mut public_key_builder = public_keys_builder
.reborrow()
.get(usize_to_u32(index).unwrap());
write_public_key(public_key, &mut public_key_builder);
}
}
My plan is to have something like this:
#[capnp_conv(funder_capnp::friends_route)]
pub struct FriendsRoute {
pub public_keys: Vec<PublicKey>,
}
Which will automatically do all the magic of producing the middle serialization layer (2), and also verify during compile time that the Rust structure has the same fields as the capnp structs. I plan to write it as a procedural macro.
Some inspiration for this plan is taken from the code I have seen in Facebook's Calibra, here: https://github.com/libra/libra/tree/master/common/proto_conv It contains a solution for a similar issue for protobuf ergonomics using a procedural macro.
I will start by writing the code as separate procedural macro crate for the Offst project. If things work out well, maybe we can add this idea into this repository's automatic code generation.
What do you think?
@realcr sounds good to me!
I have been using some of my own stuff under the hood, which boil down to basically the same thing. My setup is the following:
/// Makes the type an internal Rust representation of a Capnp.
pub trait AsCapnp {
type CapnpRepr: for<'a> Owned<'a>;
fn write_into<'a>(&self, builder: <Self::CapnpRepr as Owned<'a>>::Builder);
fn read_from<'a>(reader: <Self::CapnpRepr as Owned<'a>>::Reader) -> Result<Self, capnp::Error>
where
Self: Sized;
}
Maybe I should better split it in two traits, like Serde does, but it has been work well so far.
I indeed have tried at first to force myself to use Cap'n Proto as natively as I could, but it always ended up being better to create an internal struct anyway. Besides, I have been working with the packed representation for most purposes (I am really in for the RPC part of the deal).
Oh! I have also these two macros for stub module generation:
#[macro_export]
macro_rules! stubs {
( $( $( $declaration:ident )* ; )* ) => {
$(
_stub! { $( $declaration )* }
)*
};
}
#[macro_export]
macro_rules! _stub {
(mod $module:ident) => {
pub mod $module {
#![allow(unused)]
include!(concat!(env!("OUT_DIR"), "/", stringify!($module) , ".rs"));
}
};
(pub mod $module:ident) => {
pub mod $module {
#![allow(unused)]
include!(concat!(env!("OUT_DIR"), "/", stringify!($module) , ".rs"));
}
};
}
stubs! {
pub mod foo_capnp;
mod bar_capnp;
}
These could also be of some use.
Hi, I made some progress with the suggested plan. At this point it should be possible to automatically serialize and deserialize Rust structures using capnp.
Here is a small snippet from one of the tests, to show the syntax. Capnp file:
struct FloatStruct {
myFloat32 @0: Float32;
myFloat64 @1: Float64;
}
Rust code:
use std::convert::TryFrom;
use offst_capnp_conv::{
capnp_conv, CapnpConvError, CapnpResult, FromCapnpBytes, ReadCapnp, ToCapnpBytes, WriteCapnp,
};
#[capnp_conv(test_capnp::float_struct)]
#[derive(Debug, Clone)]
struct FloatStruct {
my_float32: f32,
my_float64: f64,
}
/// We test floats separately, because in Rust floats to not implement PartialEq
#[test]
fn capnp_serialize_floats() {
let float_struct = FloatStruct {
my_float32: -0.5f32,
my_float64: 0.5f64,
};
let data = float_struct.to_capnp_bytes().unwrap();
let float_struct2 = FloatStruct::from_capnp_bytes(&data).unwrap();
}
List(List(...))
is not yet supported.TryFrom
, CapnpResult
, ReadCapnp
, WriteCapnp
. Possibly by importing those anonymously inside the macro generated code. I'm still not sure how to do this.List(List(...))
case.I hope to complete some of those things in the following week. I also plan on trying to use capnp-conv
inside Offst, to see that it works as intended.
Currently the code is part of the Offst project. I put the code inside the Offst repository because I can't afford to wait to have it available.
@dwrensha : capnp-conv
has a license which is fully compatible with the license of capnproto-rust
(MIT || APACHE at your choice). If you decide to add the code into the capnproto-rust
project it can be added as separate two crates, without affecting the current code-base of capnproto-rust
.
Any future progress on this? I feel that the macro approach is the best path forward. The existing approach from @realcr looks extremely promising, and imo should be an officially supported way to use capnp.
The full code for the macros approach can be found here (Coded as two crates): https://github.com/freedomlayer/offst/tree/master/components/capnp_conv
You have full permission to add it into this repository. (It has exactly the same license as this repository, so no paperwork is required). I will be more than happy to have capnp_conv
as a dependency crate (:
I think that it is not perfect, and the code probably needs some polishing, but it works great for me. Hopefully it will be useful for other people.
@dwrensha: I added PR https://github.com/capnproto/capnproto-rust/pull/157, attempting to solve this issue.
I hope that the code quality of the additional crate is reasonable enough to be added to this repository. Please tell me if any modifications are required.
The code generated by capnproto-rust
is un-idiomatic, hard to use (no support for sequences, frequent u32 casts, the type-as-module that confuses IDEs, ...) and slow (bounds checks in sequences, the scratch space foot gun). Why isn't this talked about more? In most cases, the capnp types won't match the internal, domain, types anyway, so an automated macro-based lipstick won't really help.
Edit: It's much better with the improved setter ergonomics from the 0.19 version. Thank you @dwrensha!
So... I have been working with Cap'n Proto in Rust for some time by now and although I quite happy with it, I find myself writing a lot of boilerplate. How viable would it be for we to create abstractions based on Rust's macro system (thinking Serde, Rokcet, etc...) to make coding experience more fun?
Is this even the scope of any of the current crates?
I could help a bit on that.