data-plane-gateway should provision its own TLS certificates

[psFried]

We've gone a few rounds trying to use an L7 Ingress or load balancer, and it's seeming like that approach is going to be super finicky, if we can get it to work at all. The challenge here is that we're trying to support multiple protocols on a single address, and the L7 balancers we've tried will either support GRPC or HTTP, but not both at the same time.

Our proposed solution:

• Rely on an L4 load balancer to route TCP traffic to data-plane-gateway pods
• Terminate TLS at data-plane-gateway pods
• Have data-plane-gateway automatically provision and renew its TLS certificates using the autocert package

[psFried]

Here's what I'm thinking for how to test and roll out this change:

• Create a dataPlaneGatewayV2 function in data-plane-gateway.libsonnet in the ops repo, which will create the new Deployment, Service, etc.
• Deploy a second data-plane-gateway in combustible-cronut with a new domain name (?), alongside the existing one.
• Use this to test certificate provisioning and wiring up the new L4 load balancer. Test both the HTTP and GRPC services against the live brokers and reactors (read only, ofc).
• Once we're satisfied that it's working, switch the gateway url in control-plane to the new domain name.
• Remove the old data-plane-gateway resources.

@skord WDYT?

[skord]

That sounds good. We'll have to add the step in there early of reserving the static IP, probably a given, but want to write it down. We'll refer to it in dataPlaneGatewayV2. Removing the resources will include deleting the jsonnet as well as manually removing the static ip reservation in GCP.

[psFried]

Right, good call. So the updated steps would be:

• Provision a new static IP address to use with the new L4 load balancer
• Create a new DNS entry for that IP (gateway-v2.cc.estuary.dev?)
• Create a dataPlaneGatewayV2 function in data-plane-gateway.libsonnet in the ops repo, which will create the new Deployment, Service, etc.
• Deploy a second data-plane-gateway in combustible-cronut with the new domain name, alongside the existing one.
• Use this to test certificate provisioning and wiring up the new L4 load balancer. Test both the HTTP and GRPC services against the live brokers and reactors (read only, ofc).
• Once we're satisfied that it's working, switch the gateway url in control-plane to the new domain name.
• Remove the old data-plane-gateway kubernetes resources, the old jsonnet code, and the old static IP

[jgraettinger]

Can this be closed as "done"?

[skord]

@jgraettinger I think the title is misleading and we need to merge https://github.com/estuary/ops/pull/158 for cleanup before this can be considered complete. I might be wrong here, but it appears as Phil's comment 15 days ago indicates cleanup being done before it can be closed.

[skord]

estuary/ops#158 has been merged and this is closable.