Distribution of the capnp tool for Rust projects

[Bromeon]

In #181, a discussion about distributing the capnp schema generator tool (just called "tool" from now on) alongside Cap'n'Proto arised.

Status quo according to https://capnproto.org/install.html is that Windows provides binaries for download, whereas on Unix and OS X a package manager provides installations (managed by third parties). Additionally, on all platforms the tool can be built from source.

When developing a Rust application, it is common to use the crate capnpc in build.rs for schema generation. At the moment, this still requires the tool to be installed externally. This is generally acceptable, but can be a hindrance when aiming for a self-contained Rust project, e.g. for collaboration, continuous integration or deployment.

For distributing the tool, I see the following options:

1. Install separately or let the user select binaries

This is the current solution. From #181 onwards, it is possible for a Cap'n'Proto user to provide binaries for the different platforms himself and ship them along the project.

2. Build from source

build.rs would invoke CMake and build the tool from C++ source. I haven't tried this yet to see how complex the process would be.

It would require a C++ toolchain installed on the target platform however, which may not always be the case for Rust projects.

3. Ship precompiled binaries

The tool could do the same as Prost, a Rust protobuf plugin and provide executables for the most common platforms. capnpc would then select the correct one depending on the platform.

A few points would need to be considered:

• how are the executables kept up-to-date? is it required to manually recompile for all platforms?
• how often is the tool updated?
• should the tool be part of the capnpc crate? a separate crate would have several advantages:
  • binaries can be kept at their own version (often lower than capnpc one)
  • a separate Git repo avoids bloating this one with binary diffs
  • release cycles of capnpc are not tied to the binary compilation, so there would be no delay because binaries are not built yet

4. RIIR

Just mentioning this because it's bound to come up sooner or later 😬 Rewriting the tool in Rust comes at quite a cost and risk, and either will shift the problem to the C++ side, or require maintenance of two tools. I don't see this as a viable solution.

---

Obviously, besides convenience for Cap'n'Proto users, we should also try to minimize the extra maintenance needed on @dwrensha's side.

Maybe it's interesting to see how other users are using Cap'n'Proto, and what are current practices. It may turn out that this is not seen as a problem at all. Share your experiences!

[zenhack]

Right now (1) is working pretty well for me, and I don't really want to add another source of mystery binaries to my system, or have to build a large C++ project from source when I have a perfectly good distro package that works. I would be opposed to an implementation of (2) or (3) that pushed users away from using system packages when available.

A possible variation would be to provide a separate crate that bundles the executables (either by building them from source or shipping binaries, or perhaps falling back to one strategy if the other is not possible), but not have the capnpc package pull it in by default. If capnp is not available in the user's PATH, it could display an error message that included suggestions for installing the executable, one of which would be the new crate.

[Tanguille]

This open issue is interesting since I've spent a few hours today getting the capnp compiler to work with rust. It is confusing and I still haven't made it work. There is barely any information online, just the general installation guide and some examples in rust which I copied but don't work for me.

It looks like you need to install capnpc-rust. I installed both capnproto and libcapnp-dev via apt like the general install instructions state. If I do the command capnp compile -orust src/schema/user.capnp it outputs rust: no such plugin (executable should be 'capnpc-rust') rust: plugin failed: exit code 1

Only thing I can think about is that the apt repo has outdated binaries for the Cap'n Proto compiler causing it to use the old naming.

I also wanted to try it on windows since that has a totally different install method so it could work. The problem there is the instructions are so unclear I can't even imagine what I have to do. After reading it thorougly a few time it appears I don't have to do anything besides Find capnp.exe, capnpc-c++.exe, and capnpc-capnp.exe under capnproto-tools-win32-1.0.1 in the zip and copy them somewhere. since I don't care about C++ support and am not intending to use any of the .capnp files provided by the core project, or the stream keyword (for now, these instructions are also very unclear).

I really like the project conceptually and think it's mindblowing don't get me wrong but why is it so hard to install? What can I do to make it work both on my linux laptop and on my windows desktop?

[tv42]

3. Ship precompiled binaries

This tends to break in spectacular ways on less mainstream Linux boxes. musl vs glibc, nixos's /nix/store everywhere & patchelf, and so on. You'll end up with a collection of kludges, and it's just... unclean. Less trustworthy.

[tv42]

I hate to sound like a fanboy but maybe WASM can solve this -- compile the (guts of the) capnp C++ code into a WASM blob, put the file in this repo, include_bytes! it, and use some WASM engine to run it.

[tv42]

And for the sake of the conversation, here's the RIIR solution for protobuf: https://github.com/andrewhickman/protox