search

dropbox / pb-jelly

A protobuf code generation framework for the Rust language developed at Dropbox.
Apache License 2.0
611 stars 25 forks source link
dropbox protobuf rust

readme

pb-jelly

by

pb-jelly is a protobuf code generation framework for the Rust language developed at Dropbox.

History

This implementation was initially written in 2016 to satisfy the need of shuffling large amount of bytes in Dropbox's Storage System (Magic Pocket). Previously, we were using rust-protobuf (and therefore generated APIs are exactly the same to make migration easy) but serializing Rust structs to proto messages, and then serializing them again in our RPC layer, meant multiple copies (and same thing in reverse on parsing stack). Taking control of this implementation and integrating it in our RPC stack end-to-end helped avoid these extra copies.

Over the years, the implementation has grown and matured and is currently used in several parts of Dropbox, including our Sync Engine, and the aforementioned Magic Pocket.

Other implementations exist in the Rust ecosystem (e.g. prost and rust-protobuf), we wanted to share ours as well.


Crates.io Documentation Crates.io Build Status

Features


Extensions

Extension Description Type Example
(rust.zero_copy)=true Generates field type of Lazy<bytes::Bytes> for proto bytes fields to support zero-copy deserialization Field zero_copy
(rust.box_it)=true Generates a Box<Message> field type Field box_it
(rust.type)="type" Generates a custom field type Field custom_type
(rust.preserve_unrecognized)=true Preserves unrecognized proto fields into an _unrecognized struct field Field TODO
(rust.nullable_field)=false Generates non-nullable fields types Field TODO
(rust.nullable)=false Generates oneofs as non-nullable (fail on deserialization) Oneof non_optional
(rust.err_if_default_or_unknown)=true Generates enums as non-zeroable (fail on deserialization) Enum non_optional
(rust.closed_enum)=true Generates only a "closed" enum which will fail deserialization for unknown values, but is easier to work with in Rust Enum TODO
(rust.serde_derive)=true Generates serde serializable/deserializable messages File serde


Using pb-jelly in your project

Multiple crates, multiple languages, my oh my!

Essential Crates

There are only two crates you'll need: pb-jelly and pb-jelly-gen.

pb-jelly

Contains all of the important traits and structs that power our generated code, e.g. Message and Lazy. Include this as a dependency, e.g.

[dependencies]
pb-jelly = "0.0.16"
pb-jelly-gen

A framework for generating Rust structs and implementations for .proto files. In order to use pb-jelly, you need to add the pb-jelly-gen as a plugin to your protoc invocation.

We added some code here to handle the protoc invocation if you choose to use it. You'll need to add a generation crate (see examples_gen for an example) Include pb-jelly-gen as a dependency of your generation crate, and cargo run to invoke protoc for you.

[dependencies]
pb-jelly-gen = "0.0.16"

Eventually, we hope to eliminate the need for a generation crate, and simply have generation occur inside a build.rs with pb-jelly-gen as a build dependency. However https://github.com/rust-lang/cargo/issues/8709 must be resolved first.

Note that you can always invoke protoc on your own (for example if you are already doing so to generate for multiple languages) with --rust_out=codegen.py as a plugin for rust.

Generating Rust Code

  1. Install protoc, the protobuf compiler.

To generate with pb-jelly-gen

  1. Create an inner (build-step) crate which depends on pb-jelly-gen. Example
  2. cargo run in the directory of the inner generation crate

To generate manually with protoc

  1. cargo build in pb-jelly-gen
  2. protoc --plugin=protoc-gen-jellyrust=pb-jelly-gen/target/debug/protoc-gen-jellyrust --jellyrust_out=generated/ input.proto

Example

Take a look at the examples crate to see how we leverage pb-jelly-gen and build.rs to get started using protobufs in Rust!


Non-essential Crates


A Note On Scalability šŸ“

We mention "scalabilty" as a feature, what does that mean? We take an opinionated stance that every module should be a crate, as opposed to generating Rust files 1:1 with proto files. We take this stance because rustc is parallel across crates, but not yet totally parallel within a crate. When we had all of our generated Rust code in a single crate, it was often that single crate that took the longest to compile. The solution to these long compile times, was creating many crates!


The Name šŸŒ 

pb-jelly is a shoutout to the jellyfish known for its highly efficient locomotion. This library is capable of highly efficient locomotion of deserialized data. Also a shoutout to ability of the jellyfish to have substantial increases in population. This library handles generating a very large number of proto modules with complex dependencies, by generating to multiple crates.

We also like the popular sandwich.

Contributing

First, contributions are greatly appreciated and highly encouraged. For legal reasons all outside contributors must agree to Dropbox's CLA. Thank you for your understanding.


Upcoming

Some of the features here require additional tooling to be useful, which are not yet public.

Closed structs with public fields

Service Generation

Running the pbtest unit tests

  1. Clone Repo.
  2. Install Dependencies / Testing Dependencies. Use the appropriate package manager for your system.
    • protoc - part of Google's protobuf tools
      • macos: brew install protobuf
      • Linux (Fedora/CentOS/RHEL): dnf install protobuf protobuf-devel
      • Linux (Ubuntu): apt install protobuf-compiler
  3. pb-jelly currently uses an experimental test framework that requires a nightly build of rust.
    • rustup default nightly
  4. cd pb-test
  5. ( cd pb_test_gen ; cargo run ) ; cargo test

Contributors

Dropboxers [incl former]

Non-Dropbox

Similar Projects

rust-protobuf - Rust implementation of Google protocol buffers
prost - PROST! a Protocol Buffers implementation for the Rust Language
quick-protobuf - A rust implementation of protobuf parser
serde-protobuf
protokit