Deprecate API Client's usage of the Reverse Tunnel for connecting to the Auth Server

[strideynet]

The Current State

Today, when an API client is created, it tries connecting to the Auth Server in a number of ways. This accounts for the many different deployment architectures in use (e.g separate ports vs multiplexed, TLS terminating LB presence)

At the time of writing, the following connectivity pathways are posible:

• A) Connecting directly to the address as if it is an Auth Server
  • 1x TLS
  • Requires Auth Server to be directly accessible to the client.
• B) Using ALPN routing with teleport-auth@[hex-cluster-name].teleport.cluster.local to request an unterminated TLS connection through the Proxy to the Auth Server.
  • 1x TLS
  • Requires the client be configured with the name of the cluster to be used.
  • Does not work with a TLS terminating LB fronting Teleport
  • Currently incompatible with cloud - https://github.com/gravitational/teleport/issues/25534
• B+) Use the ALPN upgrade techniques to allow connection through a TLS terminating LB, then do B
  • 3x TLS ( or 2x if connection between LB and Proxy is not using TLS)
  • Requires the client be configured with the name of the cluster to be used.
• C) Using SSH to the Proxy reverse tunnel port with an address of @remote-auth-server and then creating a TLS connection within this.
  • 1x SSH, 1x TLS
  • Requires Reverse Tunnel port to be exposed
  • Unlike B, it does not require the client be configured with the cluster name.
• D) Using ALPN routing with teleport-reversetunnel, and then doing C
  • 1x SSH, 2x TLS
  • Does not work with a TLS terminating LB fronting Teleport
• D+) Use the ALPN upgrade techniques to allow connection through a TLS terminating LB, and then do D
  • 1x SSH, 4x TLS ( or 3x if connection between LB and Proxy is not using TLS)

And to visualize D+

More and more customers are using multiplexed mode instead of ports separate mode, and the published Teleport plugins (e.g Terraform/Access Plugins) do not configure the client with the name of the cluster. This means in most cases, D/D+ is being used (the most complex and least performant!)

Why change this?

Performance:

• Establishing multiple TLS/SSH connections involves many slow handshakes, increasing the amount of time to connect.
• Several layers of TLS/SSH increases resource consumption.
• The current client implementation tries connecting in a variety of ways upon creation, this worsens a thundering herd issues associated with a large number of clients crashing, and then simultaneously reconnecting.

Complexity:

• Managing connections requiring multiple layers of TLS/SSH is much more complex, increasing the risk of regressions.
• Maintaining a large variety of connection pathways, increases the work involved when adding new configuration options that should apply to all connection pathways. We often miss out support for certain features (e.g HTTP_PROXY support) in some of the connection pathways, leaving the client falling back to the other pathways.

How?

Option B/B+ is the most promising pathway for customers who need to connect through a proxy, are using multiplexed ports, and who may have a TLS terminating LB in place. It has the least number of wrapped connections and avoids the use of SSH, making it much simpler. In addition, tricks can be used to make this even more performant (e.g configuring your load balancer to detect the teleport-auth@[hex-cluster-name].teleport.cluster.local ALPN and to direct traffic directly to your Auth Server, skipping the Proxy entirely)

The main issue with B/B+ is that the client must be configured with the name of the Teleport cluster to use the teleport-auth@[hex-cluster-name].teleport.cluster.local ALPN routing. This contrasts with using the special @remote-auth-server address when using teleport-reversetunnel, which just connects you to the Auth Server the proxy is connected to.

This issue can be resolved in two different ways:

• Modify the client to read the cluster name from the credentials it is using, and configure ALPNSNIAuthDialClusterName automagically.
• Modify the ALPN router to support connecting to the Proxy's local Auth Server when a sentinel value is used e.g teleport-auth@please-connect-me-to-your-local-auth-server-thank-you. We'd also need to modify the client to use this. One advantage of this over reading the cluster name from the credentials within the client is that it'd allow us to connect to the Auth Server unauthenticated.

With the support for pathway B/B+ improved, we'd then be free to begin deprecating pathways involving the SSH reverse tunnel.

[espadolini]

If we don't have credentials it's not a big deal to try /webapi/find just the one time, and if we have credentials we know the cluster name.

Even if the agent configuration isn't using proxy_server we can just try to use /webapi/find once per process to determine whether or not we should run the agent in tunnel mode or in direct dial mode without too many issues, I think (we repeatedly hit /webapi/find for worse reasons already).

This is an also a blocker for gravitational/cloud#5006 too.