Data Plane Gateway

Gazette exposes several GRPC services we'd like to provide broader access to.

1. We'd like to expose REST versions of these services.
2. We'd like to enforce authentication and authorization.

REST Gateway

We use the grpc-gateway project to generate REST handlers from the protobuf definitions found in Gazette. We also generate Swagger documentation which we then use to generate a TypeScript client. The REST gateway forwards requests to the Auth Gateway.

Auth Gateway

The GRPC side of the Gateway proxies requests to our main Gazette cluster. First we check for an authentication token issued by the Control Plane. This token provides us with a set of authorized prefixes. We authorize requests by checking the provided label selectors on a request against these authorized prefixes. Once authorized, requests are sent along to Gazette proper.

Proxying to containers

The data plane gateway also proxies network traffic to containers of running shards. The shard and port are encoded in a subdomain label, which is used to lookup the shard and information about the ports it exposes. If the protocol associated with the port is http (h2 or http/1.1), then the gateway can optionally enforce authorization as described above. This allows shards to expose "private" http ports, which are only accessible to authorized users.

Build

To build the Gateway:

go build .
# or
go install .

Usage

The data plane gateway always requires TLS. There is no option for running without TLS. The following command launches the Gateway on port 28318. This port will be used for serving REST and GRPC handlers, as well as for proxying connections to running containers. The gateway will also listen on a non-tls port, but on this port it will only serve the health check and metrics endpoints.

To run the Gateway:

data-plane-gateway --tls-certificate /path/to/cert.pem --tls-private-key /path/to/key.pem --hostname localhost

The <hostname> must match the subject common name of the certificate, and the server must be reachable by that name. Proxy connections are always handled on a subdomain of the provided --hostname. For example, if you pass --hostname localhost, then proxy connections would be of the form 123abc-8080.localhost.

Use data-plane-gateway --help for more options.

Development

The Gateway extensively uses code generation tools. This generated code is checked into the repo, so most users don't need to worry about them.

• To Install code generation tools, you'll need a few prerequisites which you can install with make protobuf_tools.
• To run the code generation, run make.
• To hack on the Gateway, use go run ..
• To hack on the typescript client, launch a Data Plane and Gateway with ./test.sh server.
• To run all the tests, run ./test.sh run.

Components

• {broker,consumer}_service.yaml define which RPCs are exposed as REST handlers.
• gen/ houses all the files generated by grpc-gateway.
• rest.go wires up the generated REST handler code into a single server.
• journal_server.go enforces auth on top of the Gazette JournalServer interface.
• shard_server.go enforces auth on top of the Gazette ShardServer interface.
• main.go multiplexes the REST server and the GRPC server.
• auth/ decodes authentication tokens from the Control Plane and enforces our authorization rules.
• proxy/ handles both tcp and http proxying to containers running in the data plane.
• client/src defines a TypeScript library that can interact with the REST api.
• client/dist contains the generated JavaScript package.
• build.sh wrangles all the configuration options for building the gateway.
• test.sh launches a local Flow data plane, the gateway, and runs the client tests against it.