search

stephenh / ts-proto

An idiomatic protobuf generator for TypeScript
Apache License 2.0
2.12k stars 344 forks source link
dataloader grpc grpc-node grpc-web nestjs protobuf twirp typescript

readme

npm build

ts-proto

ts-proto transforms your .proto files into strongly-typed, idiomatic TypeScript files!

ts-proto 2.x Release Notes

The 2.x release of ts-proto migrated the low-level Protobuf serializing that its encode and decode method use from the venerable, but aging & stagnant, protobufjs package to @bufbuild/protobuf.

If you only used the encode and decode methods, this should largely be a non-breaking change.

However, if you used any code that used the old protobufjs Writer or Reader classes, you'll need to update your code to use the new @bufbuild/protobuf classes:

import { BinaryReader, BinaryWriter } from "@bufbuild/protobuf/wire";

If migrating to @bufbuild/protobuf is a blocker for you, you can pin your ts-proto version to 1.x.

Disclaimer & apology: I had intended to release ts-proto 2.x as an alpha release, but didn't get the semantic-release config correct, and so ts-proto 2.x was published as a major release without a proper alpha/beta cycle.

If you could file reports (or better PRs!) for any issues you come across while the release is still fresh, that would be greatly appreciated.

Any tips or tricks for others on the migration would also be appreciated!

Table of contents

Overview

ts-proto generates TypeScript types from protobuf schemas.

I.e. given a person.proto schema like:

message Person {
  string name = 1;
}

ts-proto will generate a person.ts file like:

interface Person {
  name: string
}

const Person = {
  encode(person): Writer { ... }
  decode(reader): Person { ... }
  toJSON(person): unknown { ... }
  fromJSON(data): Person { ... }
}

It also knows about services and will generate types for them as well, i.e.:

export interface PingService {
  ping(request: PingRequest): Promise<PingResponse>;
}

It will also generate client implementations of PingService; currently Twirp, grpc-web, grpc-js and nestjs are supported.

QuickStart

This will generate *.ts source files for the given *.proto types.

If you want to package these source files into an npm package to distribute to clients, just run tsc on them as usual to generate the .js/.d.ts files, and deploy the output as a regular npm package.

Buf

If you're using Buf, pass strategy: all in your buf.gen.yaml file (docs).

version: v1
plugins:
  - name: ts
    out: ../gen/ts
    strategy: all
    path: ../node_modules/ts-proto/protoc-gen-ts_proto

To prevent buf push from reading irrelevant .proto files, configure buf.yaml like so:

build:
  excludes: [node_modules]

You can also use the official plugin published to the Buf Registry.

version: v1
plugins:
  - plugin: buf.build/community/stephenh-ts-proto
    out: ../gen/ts
    opt:
      - outputServices=...
      - useExactTypes=...

ESM

If you're using a modern TS setup with either esModuleInterop or running in an ESM environment, you'll need to pass ts_proto_opts of:

Goals

In terms of the code that ts-proto generates, the general goals are:

Non-Goals

Note that ts-proto is not an out-of-the-box RPC framework; instead it's more of a swiss-army knife (as witnessed by its many config options), that lets you build exactly the RPC framework you'd like on top of it (i.e. that best integrates with your company's protobuf ecosystem; for better or worse, protobuf RPC is still a somewhat fragmented ecosystem).

If you'd like an out-of-the-box RPC framework built on top of ts-proto, there are a few examples:

(Note for potential contributors, if you develop other frameworks/mini-frameworks, or even blog posts/tutorials, on using ts-proto, we're happy to link to them.)

We also don't support clients for google.api.http-based Google Cloud APIs, see #948 if you'd like to submit a PR.

Example Types

The generated types are "just data", i.e.:

export interface Simple {
  name: string;
  age: number;
  createdAt: Date | undefined;
  child: Child | undefined;
  state: StateEnum;
  grandChildren: Child[];
  coins: number[];
}

Along with encode/decode factory methods:

export const Simple = {
  create(baseObject?: DeepPartial<Simple>): Simple {
    ...
  },

  encode(message: Simple, writer: Writer = Writer.create()): Writer {
    ...
  },

  decode(reader: Reader, length?: number): Simple {
    ...
  },

  fromJSON(object: any): Simple {
    ...
  },

  fromPartial(object: DeepPartial<Simple>): Simple {
    ...
  },

  toJSON(message: Simple): unknown {
    ...
  },
};

This allows idiomatic TS/JS usage like:

const bytes = Simple.encode({ name: ..., age: ..., ... }).finish();
const simple = Simple.decode(Reader.create(bytes));
const { name, age } = simple;

Which can dramatically ease integration when converting to/from other layers without creating a class and calling the right getters/setters.

Highlights

Auto-Batching / N+1 Prevention

(Note: this is currently only supported by the Twirp clients.)

If you're using ts-proto's clients to call backend micro-services, similar to the N+1 problem in SQL applications, it is easy for micro-service clients to (when serving an individual request) inadvertently trigger multiple separate RPC calls for "get book 1", "get book 2", "get book 3", that should really be batched into a single "get books [1, 2, 3]" (assuming the backend supports a batch-oriented RPC method).

ts-proto can help with this, and essentially auto-batch your individual "get book" calls into batched "get books" calls.

For ts-proto to do this, you need to implement your service's RPC methods with the batching convention of:

When ts-proto recognizes methods of this pattern, it will automatically create a "non-batch" version of <OperationName> for the client, i.e. client.Get<OperationName>, that takes a single id and returns a single result.

This provides the client code with the illusion that it can make individual Get<OperationName> calls (which is generally preferable/easier when implementing the client's business logic), but the actual implementation that ts-proto provides will end up making Batch<OperationName> calls to the backend service.

You also need to enable the useContext=true build-time parameter, which gives all client methods a Go-style ctx parameter, with a getDataLoaders method that lets ts-proto cache/resolve request-scoped DataLoaders, which provide the fundamental auto-batch detection/flushing behavior.

See the batching.proto file and related tests for examples/more details.

But the net effect is that ts-proto can provide SQL-/ORM-style N+1 prevention for clients calls, which can be critical especially in high-volume / highly-parallel implementations like GraphQL front-end gateways calling backend micro-services.

Usage

ts-proto is a protoc plugin, so you run it by (either directly in your project, or more likely in your mono-repo schema pipeline, i.e. like Ibotta or Namely):

protoc --plugin=node_modules/ts-proto/protoc-gen-ts_proto ./batching.proto -I.

ts-proto can also be invoked with Gradle using the protobuf-gradle-plugin:

protobuf {
    plugins {
        // `ts` can be replaced by any unused plugin name, e.g. `tsproto`
        ts {
            path = 'path/to/plugin'
        }
    }

    // This section only needed if you provide plugin options
    generateProtoTasks {
        all().each { task ->
            task.plugins {
                // Must match plugin ID declared above
                ts {
                    option 'foo=bar'
                }
            }
        }
    }
}

Generated code will be placed in the Gradle build directory.

Supported options

message ProfileInfo {
    int32 id = 1;
    string bio = 2;
    string phone = 3;
}

message Department {
    int32 id = 1;
    string name = 2;
}

message User {
    int32 id = 1;
    string username = 2;
    /*
     ProfileInfo will be optional in typescript, the type will be ProfileInfo | null | undefined
     this is needed in cases where you don't wanna provide any value for the profile.
    */
    optional ProfileInfo profile = 3;

    /*
      Department only accepts a Department type or null, so this means you have to pass it null if there is no value available.
    */
    Department  department = 4;
}

the generated interfaces will be:

export interface ProfileInfo {
  id: number;
  bio: string;
  phone: string;
}

export interface Department {
  id: number;
  name: string;
}

export interface User {
  id: number;
  username: string;
  profile?: ProfileInfo | null | undefined; // check this one
  department: Department | null; // check this one
}

This option allows the library to act in a compatible way with the Wire implementation maintained and used by Square/Block. Note: this option should only be used in combination with other client/server code generated using Wire or ts-proto with this option enabled.

NestJS Support

We have a great way of working together with nestjs. ts-proto generates interfaces and decorators for you controller, client. For more information see the nestjs readme.

Watch Mode

If you want to run ts-proto on every change of a proto file, you'll need to use a tool like chokidar-cli and use it as a script in package.json:

"proto:generate": "protoc --ts_proto_out=. ./<proto_path>/<proto_name>.proto --ts_proto_opt=esModuleInterop=true",
"proto:watch": "chokidar \"**/*.proto\" -c \"npm run proto:generate\""

Basic gRPC implementation

ts-proto is RPC framework agnostic - how you transmit your data to and from your data source is up to you. The generated client implementations all expect a rpc parameter, which type is defined like this:

interface Rpc {
  request(service: string, method: string, data: Uint8Array): Promise<Uint8Array>;
}

If you're working with gRPC, a simple implementation could look like this:

const conn = new grpc.Client(
  "localhost:8765",
  grpc.credentials.createInsecure()
);

type RpcImpl = (service: string, method: string, data: Uint8Array) => Promise<Uint8Array>;

const sendRequest: RpcImpl = (service, method, data) => {
  // Conventionally in gRPC, the request path looks like
  //   "package.names.ServiceName/MethodName",
  // we therefore construct such a string
  const path = `/${service}/${method}`;

  return new Promise((resolve, reject) => {
    // makeUnaryRequest transmits the result (and error) with a callback
    // transform this into a promise!
    const resultCallback: UnaryCallback<any> = (err, res) => {
      if (err) {
        return reject(err);
      }
      resolve(res);
    };

    function passThrough(argument: any) {
      return argument;
    }

    // Using passThrough as the serialize and deserialize functions
    conn.makeUnaryRequest(path, passThrough, passThrough, data, resultCallback);
  });
};

const rpc: Rpc = { request: sendRequest };

Sponsors

Kudos to our sponsors:

If you need ts-proto customizations or priority support for your company, you can ping me at via email.

Development

This section describes how to contribute directly to ts-proto, i.e. it's not required for running ts-proto in protoc or using the generated TypeScript.

Requirements

Setup

The commands below assume you have Docker installed. If you are using OS X, install coreutils, brew install coreutils.

Workflow

Testing in your projects

You can test your local ts-proto changes in your own projects by running yarn add ts-proto@./path/to/ts-proto, as long as you run yarn build manually.

Dockerized Protoc

The repository includes a dockerized version of protoc, which is configured in docker-compose.yml.

It can be useful in case you want to manually invoke the plugin with a known version of protoc.

Usage:

# Include the protoc alias in your shell.
. aliases.sh

# Run protoc as usual. The ts-proto directory is available in /ts-proto.
protoc --plugin=/ts-proto/protoc-gen-ts_proto --ts_proto_out=./output -I=./protos ./protoc/*.proto

# Or use the ts-protoc alias which specifies the plugin path for you.
ts-protoc --ts_proto_out=./output -I=./protos ./protoc/*.proto

Assumptions

Todo

OneOf Handling

By default, ts-proto models oneof fields "flatly" in the message, e.g. a message like:

message Foo {
  oneof either_field { string field_a = 1; string field_b = 2; }
}

Will generate a Foo type with two fields: field_a: string | undefined; and field_b: string | undefined.

With this output, you'll have to check both if object.field_a and if object.field_b, and if you set one, you'll have to remember to unset the other.

Instead, we recommend using the oneof=unions-value option, which will change the output to be an Algebraic Data Type/ADT like:

interface YourMessage {
  eitherField?: { $case: "field_a"; value: string } | { $case: "field_b"; value: string };
}

As this will automatically enforce only one of field_a or field_b "being set" at a time, because the values are stored in the eitherField field that can only have a single value at a time.

(Note that eitherField is optional b/c oneof in Protobuf means "at most one field" is set, and does not mean one of the fields must be set.)

In ts-proto's currently-unscheduled 2.x release, oneof=unions-value will become the default behavior.

There is also a oneof=unions option, which generates a union where the field names are included in each option:

interface YourMessage {
  eitherField?: { $case: "field_a"; field_a: string } | { $case: "field_b"; field_b: string };
}

This is no longer recommended as it can be difficult to write code and types to handle multiple oneof options:

OneOf Type Helpers

The following helper types may make it easier to work with the types generated from oneof=unions, though they are generally not needed if you use oneof=unions-value:

/** Extracts all the case names from a oneOf field. */
type OneOfCases<T> = T extends { $case: infer U extends string } ? U : never;

/** Extracts a union of all the value types from a oneOf field */
type OneOfValues<T> = T extends { $case: infer U extends string; [key: string]: unknown } ? T[U] : never;

/** Extracts the specific type of a oneOf case based on its field name */
type OneOfCase<T, K extends OneOfCases<T>> = T extends {
  $case: K;
  [key: string]: unknown;
}
  ? T
  : never;

/** Extracts the specific type of a value type from a oneOf field */
type OneOfValue<T, K extends OneOfCases<T>> = T extends {
  $case: infer U extends K;
  [key: string]: unknown;
}
  ? T[U]
  : never;

For comparison, the equivalents for oneof=unions-value:

/** Extracts all the case names from a oneOf field. */
type OneOfCases<T> = T['$case'];

/** Extracts a union of all the value types from a oneOf field */
type OneOfValues<T> = T['value'];

/** Extracts the specific type of a oneOf case based on its field name */
type OneOfCase<T, K extends OneOfCases<T>> = T extends {
  $case: K;
  [key: string]: unknown;
}
  ? T
  : never;

/** Extracts the specific type of a value type from a oneOf field */
type OneOfValue<T, K extends OneOfCases<T>> = T extends {
  $case: infer U extends K;
  value: unknown;
}
  ? T[U]
  : never;

Default values and unset fields

In core Protobuf (and so also ts-proto), values that are unset or equal to the default value are not sent over the wire.

For example, the default value of a message is undefined. Primitive types take their natural default value, e.g. string is '', number is 0, etc.

Protobuf chose/enforces this behavior because it enables forward compatibility, as primitive fields will always have a value, even when omitted by outdated agents.

This is good, but it also means default and unset values cannot be distinguished in ts-proto fields; it's just fundamentally how Protobuf works.

If you need primitive fields where you can detect set/unset, see Wrapper Types.

Encode / Decode

ts-proto follows the Protobuf rules, and always returns default values for unsets fields when decoding, while omitting them from the output when serialized in binary format.

syntax = "proto3";
message Foo {
  string bar = 1;
}
protobufBytes; // assume this is an empty Foo object, in protobuf binary format
Foo.decode(protobufBytes); // => { bar: '' }
Foo.encode({ bar: "" }); // => { }, writes an empty Foo object, in protobuf binary format

fromJSON / toJSON

Reading JSON will also initialize the default values. Since senders may either omit unset fields, or set them to the default value, use fromJSON to normalize the input.

Foo.fromJSON({}); // => { bar: '' }
Foo.fromJSON({ bar: "" }); // => { bar: '' }
Foo.fromJSON({ bar: "baz" }); // => { bar: 'baz' }

When writing JSON, ts-proto normalizes messages by omitting unset fields and fields set to their default values.

Foo.toJSON({}); // => { }
Foo.toJSON({ bar: undefined }); // => { }
Foo.toJSON({ bar: "" }); // => { } - note: omitting the default value, as expected
Foo.toJSON({ bar: "baz" }); // => { bar: 'baz' }

Well-Known Types

Protobuf comes with several predefined message definitions, called "Well-Known Types". Their interpretation is defined by the Protobuf specification, and libraries are expected to convert these messages to corresponding native types in the target language.

ts-proto currently automatically converts these messages to their corresponding native types.

Wrapper Types

Wrapper Types are messages containing a single primitive field, and can be imported in .proto files with import "google/protobuf/wrappers.proto".

Since these are messages, their default value is undefined, allowing you to distinguish unset primitives from their default values, when using Wrapper Types. ts-proto generates these fields as <primitive> | undefined.

For example:

// Protobuf
syntax = "proto3";

import "google/protobuf/wrappers.proto";

message ExampleMessage {
  google.protobuf.StringValue name = 1;
}
// TypeScript
interface ExampleMessage {
  name: string | undefined;
}

When encoding a message the primitive value is converted back to its corresponding wrapper type:

ExampleMessage.encode({ name: "foo" }); // => { name: { value: 'foo' } }, in binary

When calling toJSON, the value is not converted, because wrapper types are idiomatic in JSON.

ExampleMessage.toJSON({ name: "foo" }); // => { name: 'foo' }

JSON Types (Struct Types)

Protobuf's language and types are not sufficient to represent all possible JSON values, since JSON may contain values whose type is unknown in advance. For this reason, Protobuf offers several additional types to represent arbitrary JSON values.

These are called Struct Types, and can be imported in .proto files with import "google/protobuf/struct.proto".

ts-proto automatically converts back and forth between these Struct Types and their corresponding JSON types.

Example:

// Protobuf
syntax = "proto3";

import "google/protobuf/struct.proto";

message ExampleMessage {
  google.protobuf.Value anything = 1;
}
// TypeScript
interface ExampleMessage {
  anything: any | undefined;
}

Encoding a JSON value embedded in a message, converts it to a Struct Type:

ExampleMessage.encode({ anything: { name: "hello" } });
/* Outputs the following structure, encoded in protobuf binary format:
{
  anything: Value {
    structValue = Struct {
      fields = [
        MapEntry {
          key = "name",
          value = Value {
            stringValue = "hello"
          }
        ]
      }
    }
 }
}*/

ExampleMessage.encode({ anything: true });
/* Outputs the following structure encoded in protobuf binary format:
{
  anything: Value {
    boolValue = true
  }
}*/

Timestamp

The representation of google.protobuf.Timestamp is configurable by the useDate flag. The useJsonTimestamp flag controls precision when useDate is false.

Protobuf well-known type Default/useDate=true useDate=false useDate=string useDate=string-nano
google.protobuf.Timestamp Date { seconds: number, nanos: number } string string

When using useDate=false and useJsonTimestamp=raw timestamp is represented as { seconds: number, nanos: number }, but has nanosecond precision.

When using useDate=string-nano timestamp is represented as an ISO string with nanosecond precision 1970-01-01T14:27:59.987654321Z and relies on nano-date library for conversion. You'll need to install it in your project.

Number Types

Numbers are by default assumed to be plain JavaScript numbers.

This is fine for Protobuf types like int32 and float, but 64-bit types like int64 can't be 100% represented by JavaScript's number type, because int64 can have larger/smaller values than number.

ts-proto's default configuration (which is forceLong=number) is to still use number for 64-bit fields, and then throw an error if a value (at runtime) is larger than Number.MAX_SAFE_INTEGER.

If you expect to use 64-bit / higher-than-MAX_SAFE_INTEGER values, then you can use the ts-proto forceLong option, which uses the long npm package to support the entire range of 64-bit values.

The protobuf number types map to JavaScript types based on the forceLong config option:

Protobuf number types Default/forceLong=number forceLong=long forceLong=string
double number number number
float number number number
int32 number number number
int64 number* Long string
uint32 number number number
uint64 number* Unsigned Long string
sint32 number number number
sint64 number* Long string
fixed32 number number number
fixed64 number* Unsigned Long string
sfixed32 number number number
sfixed64 number* Long string

Where (*) indicates they might throw an error at runtime.

Current Status of Optional Values