Phil/connecting connectors

Discussion: #840 Issue: #868

Introduces support for network access to connector containers. This commit goes along with: estaury/flow@7f7c2593ac94ed5703b70e99b06c03ea8f053460 See that flow commit message for a high-level overview.

This commit adds support for proxying network connections in DPG. There's a fair bit of refactoring that was required in order to get this working, and that's also included in this commit.

When DPG accets a connection on the TLS port, it checks the Server Name Indicator (SNI) in the TLS client hello message to see if the connection attempt is for a subdomain of its hostname. If it is, then the connection attempt is treated as though the client wishes to connect to a connector container. If so, then DPG attempts to resolve the hostname to a running shard, to which it will proxy traffic. The shard resolution is done as part of the TLS handshake, so that protocol configuration from the shard can be used to complete ALPN. Once a successful tls handshake has completed, DPG will handle the proxy either as HTTP or as plain TCP, depending on the negotiated protocol.

If HTTP (either 1.1 or 2) is to be used, then DPG will handle the connection using an L7 HTTP proxy server. It will also enforce token-based authorization unless the port is explicitly marked as public.

If the protocol is anything but http/1.1 or h2, then DPG will require that the port visiblity is public, and close the connection if not. It will proxy the traffic as plain TCP (L4). This allows connectors to use arbitrary TCP-based protocols such as Kafka, Postgres, etc.

No more autocert. The network proxying in DPG requires the use of a wildcard TLS certificate. While it is possible to automatically provision a TLS certificate using the ACME protocol, doing so requires capability and permissions to programatically manage DNS records for the DPG domain. This seemed impractical to require within DPG itself, and thus TLS certificate provisioning will move into another component. DPG now simply expects its TLS certificates to exist at paths specified in the arguments.

The second commit adds a simple JSON REST handler that serves information on exposed ports of tasks. Clients can send a GET request to /endpoints/my/task/name and receive back a JSON response with the hostnames and other information about the exposed ports of a task. An example response is:

{
  "endpoints": [
    {
      "host": "123abc-7777.data-plane.estuary.data.dev",
      "protocols": [
        "h2",
        "http/1.1"
      ],
      "public": false
    },
    {
      "host": "123abc-8888.data-plane.estuary.data.dev",
      "protocols": [
        "http/1.1"
      ],
      "public": true
    }
  ]
}

estuary / data-plane-gateway

Phil/connecting connectors #27