search

bufbuild / protoc-gen-validate

Protocol Buffer Validation - Being replaced by github.com/bufbuild/protovalidate
https://github.com/bufbuild/protovalidate
Apache License 2.0
3.81k stars 580 forks source link
constraints protoc protoc-plugin protocol-buffers validation

readme

protoc-gen-validate (PGV)

License Release Slack

[!IMPORTANT] protoc-gen-validate (PGV) has reached a stable state and is in maintenance mode.

We recommend that new and existing projects transition to using protovalidate.

Read our blog post if you want to learn more about the limitations of protoc-gen-validate and how we have designed protovalidate to be better.

PGV is a protoc plugin to generate polyglot message validators. While protocol buffers effectively guarantee the types of structured data, they cannot enforce semantic rules for values. This plugin adds support to protoc-generated code to validate such constraints.

Developers import the PGV extension and annotate the messages and fields in their proto files with constraint rules:

syntax = "proto3";

package examplepb;

import "validate/validate.proto";

message Person {
  uint64 id = 1 [(validate.rules).uint64.gt = 999];

  string email = 2 [(validate.rules).string.email = true];

  string name = 3 [(validate.rules).string = {
    pattern:   "^[A-Za-z]+( [A-Za-z]+)*$",
    max_bytes: 256,
  }];

  Location home = 4 [(validate.rules).message.required = true];

  message Location {
    double lat = 1 [(validate.rules).double = {gte: -90,  lte: 90}];
    double lng = 2 [(validate.rules).double = {gte: -180, lte: 180}];
  }
}

Executing protoc with PGV and the target language's default plugin will create Validate methods on the generated types:

p := new(Person)

err := p.Validate() // err: Id must be greater than 999
p.Id = 1000

err = p.Validate() // err: Email must be a valid email address
p.Email = "example@bufbuild.com"

err = p.Validate() // err: Name must match pattern '^[A-Za-z]+( [A-Za-z]+)*$'
p.Name = "Protocol Buffer"

err = p.Validate() // err: Home is required
p.Home = &Location{37.7, 999}

err = p.Validate() // err: Home.Lng must be within [-180, 180]
p.Home.Lng = -122.4

err = p.Validate() // err: nil

Usage

Dependencies

Installation

Download from GitHub Releases

Download assets from GitHub Releases and unarchive them and add plugins into $PATH.

Build from source

# fetches this repo into $GOPATH
go get -d github.com/envoyproxy/protoc-gen-validate

💡 Yes, our go module path is github.com/envoyproxy/protoc-gen-validate not bufbuild this is intentional.

Changing the module path is effectively creating a new, independent module. We would prefer not to break our users. The Go team are working on better cmd/go support for modules that change paths, but progress is slow. Until then, we will continue to use the envoyproxy module path.

git clone https://github.com/bufbuild/protoc-gen-validate.git
# installs PGV into $GOPATH/bin
cd protoc-gen-validate && make build

Parameters

Examples

Go

Go generation should occur into the same output path as the official plugin. For a proto file example.proto, the corresponding validation code is generated into ../generated/example.pb.validate.go:

protoc \
  -I . \
  -I path/to/validate/ \
  --go_out=":../generated" \
  --validate_out="lang=go:../generated" \
  example.proto

All messages generated include the following methods:

PGV requires no additional runtime dependencies from the existing generated code.

Note: by default example.pb.validate.go is nested in a directory structure that matches your option go_package name. You can change this using the protoc parameter paths=source_relative:., as like --validate_out="lang=go,paths=source_relative:../generated". Then --validate_out will output the file where it is expected. See Google's protobuf documentation or packages and input paths or parameters for more information.

There's also support for the module=example.com/foo flag described here .

With newer Buf CLI versions (>v1.9.0), you can use the new plugin key instead of using the protoc command directly:

# buf.gen.yaml

version: v1
plugins:
  - plugin: buf.build/bufbuild/validate-go
    out: gen
# proto/buf.yaml

version: v1
deps:
  - buf.build/envoyproxy/protoc-gen-validate

Java

Java generation is integrated with the existing protobuf toolchain for java projects. For Maven projects, add the following to your pom.xml or build.gradle.


<dependencies>
    <dependency>
        <groupId>build.buf.protoc-gen-validate</groupId>
        <artifactId>pgv-java-stub</artifactId>
        <version>${pgv.version}</version>
    </dependency>
</dependencies>

<build>
<extensions>
    <extension>
        <groupId>kr.motd.maven</groupId>
        <artifactId>os-maven-plugin</artifactId>
        <version>1.4.1.Final</version>
    </extension>
</extensions>
<plugins>
    <plugin>
        <groupId>org.xolstice.maven.plugins</groupId>
        <artifactId>protobuf-maven-plugin</artifactId>
        <version>0.6.1</version>
        <configuration>
            <protocArtifact>
                com.google.protobuf:protoc:${protoc.version}:exe:${os.detected.classifier}
            </protocArtifact>
        </configuration>
        <executions>
            <execution>
                <id>protoc-java-pgv</id>
                <goals>
                    <goal>compile-custom</goal>
                </goals>
                <configuration>
                    <pluginParameter>lang=java</pluginParameter>
                    <pluginId>java-pgv</pluginId>
                    <pluginArtifact>
                        build.buf.protoc-gen-validate:protoc-gen-validate:${pgv.version}:exe:${os.detected.classifier}
                    </pluginArtifact>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>
</build>
plugins {
    ...
    id "com.google.protobuf" version "${protobuf.version}"
    ...
}

protobuf {
    protoc {
        artifact = "com.google.protobuf:protoc:${protoc.version}"
    }

    plugins {
        javapgv {
            artifact = "build.buf.protoc-gen-validate:protoc-gen-validate:${pgv.version}"
        }
    }

    generateProtoTasks {
        all()*.plugins {
            javapgv {
                option "lang=java"
            }
        }
    }
}
// Create a validator index that reflectively loads generated validators
ValidatorIndex index = new ReflectiveValidatorIndex();
// Assert that a message is valid
index.validatorFor(message.getClass()).assertValid(message);

// Create a gRPC client and server interceptor to automatically validate messages (requires pgv-java-grpc module)
clientStub = clientStub.withInterceptors(new ValidatingClientInterceptor(index));
serverBuilder.addService(ServerInterceptors.intercept(svc, new ValidatingServerInterceptor(index)));

Python

The python implementation works via JIT code generation. In other words, the validate(msg) function is written on-demand and exec-ed. An LRU-cache improves performance by storing generated functions per descriptor.

The python package is available on PyPI.

To run validate(), do the following:

from entities_pb2 import Person
from protoc_gen_validate.validator import validate, ValidationFailed

p = Person(first_name="Foo", last_name="Bar", age=42)
try:
    validate(p)
except ValidationFailed as err:
    print(err)

You can view what code has been generated by using the print_validate() function.

Constraint Rules

The provided constraints are modeled largerly after those in JSON Schema. PGV rules can be mixed for the same field; the plugin ensures the rules applied to a field cannot contradict before code generation.

Check the constraint rule comparison matrix for language-specific constraint capabilities.

Numerics

All numeric types (float, double, int32, int64, uint32, uint64 , sint32, sint64, fixed32, fixed64, sfixed32, sfixed64) share the same rules.

Bools

Strings

Bytes

Literal values should be expressed with strings, using escaping where necessary.

Enums

All literal values should use the numeric (int32) value as defined in the enum descriptor.

The following examples use this State enum

enum State {
  INACTIVE = 0;
  PENDING = 1;
  ACTIVE = 2;
}

Messages

If a field contains a message and the message has been generated with PGV, validation will be performed recursively. Message's not generated with PGV are skipped.

// if Person was generated with PGV and x is set,
// x's fields will be validated.
    Person x = 1;

Repeated

Maps

Well-Known Types (WKTs)

A set of WKTs are packaged with protoc and common message patterns useful in many domains.

Scalar Value Wrappers

In the proto3 syntax, there is no way of distinguishing between unset and the zero value of a scalar field. The value WKTs permit this differentiation by wrapping them in a message. PGV permits using the same scalar rules that the wrapper encapsulates.

// if it is set, x must be greater than 3
    google.protobuf.Int32Value x = 1 [(validate.rules).int32.gt = 3];

Message Rules can also be used with scalar Well-Known Types (WKTs):

// Ensures that if a value is not set for age, it would not pass the validation despite its zero value being 0.
message X {google.protobuf.Int32Value age = 1 [(validate.rules).int32.gt = -1, (validate.rules).message.required = true];}

Anys

Durations

Timestamps

Message-Global

OneOfs

Development

PGV is written in Go on top of the protoc-gen-star framework and compiles to a standalone binary.

Dependencies

All PGV dependencies are currently checked into the project. To test PGV, protoc must be installed, either from source, the provided releases, or a package manager. The official protoc plugin for the target language(s) should be installed as well.

Make Targets

Run all tests under Bazel

Ensure that your PATH is setup to include protoc-gen-go and protoc, then:

bazel test //tests/...

Docker

PGV comes with a Dockerfile for consistent development tooling and CI. The main entrypoint is make with build as the default target.

# build the image
docker build -t bufbuild/protoc-gen-validate .

# executes the default make target: build
docker run --rm \
  bufbuild/protoc-gen-validate

# executes the 'ci' make target
docker run --rm \
  bufbuild/protoc-gen-validate ci

# executes the 'build' & 'testcases' make targets
docker run --rm \
  bufbuild/protoc-gen-validate build testcases

# override the entrypoint and interact with the container directly
# this can be useful when wanting to run bazel commands without
# bazel installed locally.
docker run --rm \
 -it --entrypoint=/bin/bash \
 bufbuild/protoc-gen-validate