Protocol Buffers for Elixir

exprotobuf works by building module/struct definitions from a Google Protocol Buffer schema. This allows you to work with protocol buffers natively in Elixir, with easy decoding/encoding for transport across the wire.

Features

• Load protobuf from file or string
• Respects the namespace of messages
• Allows you to specify which modules should be loaded in the definition of records
• Currently uses gpb for protobuf schema parsing

TODO:

• Clean up code/tests

Breaking Changes

The 1.0 release removed the feature of handling import "..."; statements. Please see the imports upgrade guide for details if you were using this feature.

Getting Started

Add exprotobuf as a dependency to your project:

defp deps do
  [{:exprotobuf, "~> x.x.x"}]
end

Then run mix deps.get to fetch.

Add exprotobuf to applications list:

def application do
  [applications: [:exprotobuf]]
end

Usage

Usage of exprotobuf boils down to a single use statement within one or more modules in your project.

Let's start with the most basic of usages:

Define from a string

defmodule Messages do
  use Protobuf, """
    message Msg {
      message SubMsg {
        required uint32 value = 1;
      }

      enum Version {
        V1 = 1;
        V2 = 2;
      }

      required Version version = 2;
      optional SubMsg sub = 1;
    }
  """
end

iex> msg = Messages.Msg.new(version: :'V2')
%Messages.Msg{version: :V2, sub: nil}
iex> encoded = Messages.Msg.encode(msg)
<<16, 2>>
iex> Messages.Msg.decode(encoded)
%Messages.Msg{version: :V2, sub: nil}

The above code takes the provided protobuf schema as a string, and generates modules/structs for the types it defines. In this case, there would be a Msg module, containing a SubMsg and Version module. The properties defined for those values are keys in the struct belonging to each. Enums do not generate structs, but a specialized module with two functions: atom(x) and value(x). These will get either the name of the enum value, or it's associated value.

Values defined in the schema using the oneof construct are represented with tuples:

defmodule Messages do
  use Protobuf, """
    message Msg {
      oneof choice {
        string first = 1;
        int32 second = 2;
      }
    }
  """
end

iex> msg = Messages.Msg.new(choice: {:second, 42})
%Messages.Msg{choice: {:second, 42}}
iex> encoded = Messages.Msg.encode(msg)
<<16, 42>>

Define from a file

defmodule Messages do
  use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__)
end

This is equivalent to the above, if you assume that messages.proto contains the same schema as in the string of the first example.

Loading all definitions from a set of files

defmodule Protobufs do
  use Protobuf, from: Path.wildcard(Path.expand("../definitions/**/*.proto", __DIR__))
end

iex> Protobufs.Msg.new(v: :V1)
%Protobufs.Msg{v: :V1}
iex> %Protobufs.OtherMessage{middle_name: "Danger"}
%Protobufs.OtherMessage{middle_name: "Danger"}

This will load all the various definitions in your .proto files and allow them to share definitions like enums or messages between them.

Customizing Generated Module Names

In some cases your library of protobuf definitions might already contain some namespaces that you would like to keep. In this case you will probably want to pass the use_package_names: true option. Let's say you had a file called protobufs/example.proto that contained:

package world;
message Example {
  enum Continent {
    ANTARCTICA = 0;
    EUROPE = 1;
  }

  optional Continent continent = 1;
  optional uint32 id = 2;
}

You could load that file (and everything else in the protobufs directory) by doing:

defmodule Definitions do
  use Protobuf, from: Path.wildcard("protobufs/*.proto"), use_package_names: true
end

iex> Definitions.World.Example.new(continent: :EUROPE)
%Definitions.World.Example{continent: :EUROPE}

You might also want to define all of these modules in the top-level namespace. You can do this by passing an explicit namespace: :"Elixir" option.

defmodule Definitions do
  use Protobuf, from: Path.wildcard("protobufs/*.proto"),
                use_package_names: true,
                namespace: :"Elixir"
end

iex> World.Example.new(continent: :EUROPE)
%World.Example{continent: :EUROPE}

Now you can use just the package names and message names that your team is already familiar with.

Inject a definition into an existing module

This is useful when you only have a single type, or if you want to pull the module definition into the current module instead of generating a new one.

defmodule Msg do
  use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__), inject: true

  def update(msg, key, value), do: Map.put(msg, key, value)
end

iex> %Msg{}
%Msg{v: :V1}
iex> Msg.update(%Msg{}, :v, :V2)
%Msg{v: :V2}

As you can see, Msg is no longer created as a nested module, but is injected right at the top level. I find this approach to be a lot cleaner than use_in, but may not work in all use cases.

Inject a specific type from a larger subset of types

When you have a large schema, but perhaps only care about a small subset of those types, you can use :only:

defmodule Messages do
  use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__),
only: [:TypeA, :TypeB]
end

Assuming that the provided .proto file contains multiple type definitions, the above code would extract only TypeA and TypeB as nested modules. Keep in mind your dependencies, if you select a child type which depends on a parent, or another top-level type, exprotobuf may fail, or your code may fail at runtime.

You may only combine :only with :inject when :only is a single type, or a list containing a single type. This is due to the restriction of one struct per module. Theoretically you should be able to pass :only with multiple types, as long all but one of the types is an enum, since enums are just generated as modules, this does not currently work though.

Extend generated modules via use_in

If you need to add behavior to one of the generated modules, use_in will help you. The tricky part is that the struct for the module you use_in will not be defined yet, so you can't rely on it in your functions. You can still work with the structs via the normal Maps API, but you lose compile-time guarantees. I would recommend favoring :inject over this when possible, as it's a much cleaner solution.

defmodule Messages do
  use Protobuf, "
    message Msg {
      enum Version {
        V1 = 1;
        V2 = 1;
      }
      required Version v = 1;
    }
  "

  defmodule MsgHelpers do
    defmacro __using__(_opts) do
      quote do
        def convert_to_record(msg) do
          msg
          |> Map.to_list
          |> Enum.reduce([], fn {_key, value}, acc -> [value | acc] end)
          |> Enum.reverse
          |> list_to_tuple
        end
      end
    end
  end

  use_in "Msg", MsgHelpers
end

iex> Messages.Msg.new |> Messages.Msg.convert_to_record
{Messages.Msg, :V1}

Attribution/License

exprotobuf is a fork of the azukiaapp/elixir-protobuf project, both of which are released under Apache 2 License.

Check LICENSE files for more information.