Stop using auth server IPs/ports anywhere in favour of proxy hostnames/ports

[webvictim]

What

Teleport currently supports 5 different services/protocols:

• Server access (ssh_service)
• Kubernetes access (kubernetes_service)
• Database access (db_service)
• Application access (app_service)
• Windows Desktop access (windows_desktop_service)

There is also one extra internal service which can be deployed separately:

• Proxy connections to the auth service (proxy_service)

Of these, only proxy_service and ssh_service can be joined directly to the auth service (port 3025) when run in a separate process.

• For proxy_service, this is required because the proxy service is responsible for facilitating and maintaining reverse tunnels. It is the public-facing proxy/bastion part of Teleport and requires GRPC communication with the auth_service as its stateful backend storage.
• For ssh_service, this is maintained due to backwards compatibility and historical use cases where reverse tunnelling was not implemented for agents.
  • It also scales better when deploying very large numbers of agents (>10k), as there is less reverse tunnel overhead.
  • However, the ssh_service can also join via reverse tunnel ("IoT mode") and this method is more useful for the majority of users as it doesn't require them to open a port (default 3022) on nodes.

All the other services listed must be joined via the proxy server (port 3080, or more commonly 443 when deployed behind load balancers with Helm/Terraform) as they use reverse tunnels for their connectivity.

We do not expose the auth service port (3025) in our example Helm or Terraform configurations, mainly for security. The same is true for Teleport Cloud. Although the auth service has very few unauthenticated endpoints and all requests generally require authentication via mTLS to succeed, there is just no need to have the auth service exposed to the outside world in 99% of cases, given that only the proxy_service requires a connection to it (and this is commonly run in the same process as the auth service)

Given all this knowledge, it is counterproductive that tctl tokens add (for type node, kube or windows_desktop) and other variants (tctl nodes add) list the auth server's IP and port in their example commands:

Node:

~ » tctl tokens add --type=node
The invite token: cb280b78213e81b5e1ead4c71b9a106d.
This token will expire in 60 minutes.

Run this on the new node to join the cluster:

> teleport start \
   --roles=node \
   --token=cb280b78213e81b5e1ead4c71b9a106d \
   --ca-pin=sha256:015c54b353210befba6e36480f0db89541b7a94d088c0cea21ffc2ea27848799 \
   --auth-server=172.30.0.1:3025

Please note:

  - This invitation token will expire in 60 minutes
  - 172.30.0.1:3025 must be reachable from the new node

Kube:

~ » tctl tokens add --type=kube
The invite token: 4ef0795e3cfe6792bcf4209f44f30cd7.
This token will expire in 60 minutes.

Run this on the new node to join the cluster:

> teleport start \
   --roles=kube \
   --token=4ef0795e3cfe6792bcf4209f44f30cd7 \
   --ca-pin=sha256:015c54b353210befba6e36480f0db89541b7a94d088c0cea21ffc2ea27848799 \
   --auth-server=172.30.0.1:3025

Please note:

  - This invitation token will expire in 60 minutes
  - 172.30.0.1:3025 must be reachable from the new node

Windows desktop:

~ » tctl tokens add --type=windows_desktop
The invite token: 23bff56870a0934dbb90c6138095482a.
This token will expire in 60 minutes.

Run this on the new node to join the cluster:

> teleport start \
   --roles=windowsdesktop \
   --token=23bff56870a0934dbb90c6138095482a \
   --ca-pin=sha256:015c54b353210befba6e36480f0db89541b7a94d088c0cea21ffc2ea27848799 \
   --auth-server=172.30.0.1:3025

Please note:

  - This invitation token will expire in 60 minutes
  - 172.30.0.1:3025 must be reachable from the new node

The problems here are manyfold:

• 172.30.0.1 is an internal IP, not routable from the internet
• Even if this IP were routable, port 3025 is not open, so communication would fail.

Instead, these messages should use the proxy service's public_addr and port as an example. The app and db types already do this.

Related to #11457 (because using teleport start in these messages is also something we should stop doing)

How

Code changes.

Why

The default outputs are just not useful to the majority of users with modern deployments, and in most cases are actively harming people's ability to successfully deploy Teleport without intervention from support teams.

[zmb3]

All the other services listed must be joined via the proxy server

I'm pretty sure this is not true. Desktop service and Kube service should support joining via auth.