Failing teleport-kube-agent join to cluster gives misleading error on bad token

Description

What happened: When trying to join a kubernetes cluster to an existing Teleport environment, if you use an invalid/ incorrect token, the K8S kube-agent pod does not full initialise, and reports the error:

Kube failed to establish connection to cluster: access denied, Post "https://teleport.cluster.local/v2/tokens/register": remote error: tls: internal error. service/connect.go:70

The TelePort cluster reports a certificate error:

2021-08-03T15:20:39Z ERRO [PROXY:SER] "proxy2021/08/03 15:20:39 http: TLS handshake error from x.x.x.x:1823: acme can't get a cert for domain teleport.cluster.local, add it to the proxy_service.public_addr, or use one of the domains: <hostname>.<domain>.org\n" utils/cli.go:304

This is misleading as suggests the problem is with the certificate.

What you expected to happen: Ideally the error message logged on the K8S pod should state that the token is invalid, where it is visible to the end user configuring the pod.

Reproduction Steps

As minimally and precisely as possible, describe step-by-step how to reproduce the problem.

Inintialise a new Teleport cluster
Deploy the teleport-kube-agent helm chart @ https://charts.releases.teleport.dev/ with an incorrect token (I used base64 format)
Note that once the agent pod has successfully connected to the cluster, any subsequent attempts correctly report an incorrect token. This appears to be for the initial connection.

Server Details

Teleport version (run teleport version): Teleport Enterprise v6.2.8
Server OS (e.g. from /etc/os-release): Ubuntu 20.04.2 LTS
Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS
Additional details:

Client Details

Tsh version (tsh version):
Computer OS (e.g. Linux, macOS, Windows): Ubuntu 20.04.2 LTS
Browser version (for UI-related issues):
Installed via (e.g. apt, yum, brew, website download): helm chart @ https://charts.releases.teleport.dev/
Additional details: K8S version: v1.21.3

Debug Logs

Please include or attach debug logs, when appropriate. Obfuscate sensitive information!

Start Teleport with --debug flag (teleport --debug)
Run tsh with --debug flag (tsh --debug)

Similar error occurs when node is trying to join the cluster with an unknown token:

# teleport version
Teleport v6.1.3 git: go1.16.8
# teleport start --roles=node --nodename=my-node-name --token=unregistered-token --auth-server=teleport.XXX.XXX:443
[...]
INFO [PROC:1] Joining the cluster with a secure token. service/connect.go:353
ERRO [PROC:1] Node failed to establish connection to cluster: access denied, Post "https://teleport.cluster.local/v2/tokens/register": remote error: tls: internal error. service/connect.go:67

and on server side (running Teleport 7.3.6):

teleport-7f79f44f49-rs8jl teleport 2021-12-08T12:58:20Z INFO [AUTH]      Node "my-node-name" [0d44a291-1fef-4494-9460-19ab4a428323] is trying to join with role: Node. auth/auth.go:1711
teleport-7f79f44f49-rs8jl teleport 2021-12-08T12:58:20Z WARN [AUTH]      "my-node-name" [0d44a291-1fef-4494-9460-19ab4a428323] can not join the cluster with role Node, token error: provisioning token(***************-token) not found auth/auth.go:1727
teleport-7f79f44f49-rs8jl teleport 2021-12-08T12:58:20Z WARN [MXWEB]     Handshake failed. error:[
teleport-7f79f44f49-rs8jl teleport ERROR REPORT:
teleport-7f79f44f49-rs8jl teleport Original Error: *trace.BadParameterError acme can&#39;t get a cert for domain teleport.cluster.local, add it to the proxy_service.public_addr, or use one of the domains: teleport.XXX.XXX
teleport-7f79f44f49-rs8jl teleport Stack Trace:
teleport-7f79f44f49-rs8jl teleport      /go/src/github.com/gravitational/teleport/lib/service/acme.go:70 github.com/gravitational/teleport/lib/service.(*hostPolicyChecker).checkHost
teleport-7f79f44f49-rs8jl teleport      /go/src/github.com/gravitational/teleport/vendor/golang.org/x/crypto/acme/autocert/autocert.go:301 golang.org/x/crypto/acme/autocert.(*Manager).GetCertificate
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/common.go:1017 crypto/tls.(*Config).getCertificate
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/handshake_server_tls13.go:377 crypto/tls.(*serverHandshakeStateTLS13).pickCertificate
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/handshake_server_tls13.go:52 crypto/tls.(*serverHandshakeStateTLS13).handshake
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/handshake_server.go:51 crypto/tls.(*Conn).serverHandshake
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/conn.go:1391 crypto/tls.(*Conn).Handshake
teleport-7f79f44f49-rs8jl teleport      /go/src/github.com/gravitational/teleport/lib/multiplexer/web.go:135 github.com/gravitational/teleport/lib/multiplexer.(*WebListener).detectAndForward
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/runtime/asm_amd64.s:1371 runtime.goexit
teleport-7f79f44f49-rs8jl teleport User Message: acme can&#39;t get a cert for domain teleport.cluster.local, add it to the proxy_service.public_addr, or use one of the domains: teleport.XXX.XXX] multiplexer/web.go:137
teleport-7f79f44f49-rs8jl teleport 2021-12-08T12:58:20Z WARN [MXWEB]     Handshake failed. error:[
teleport-7f79f44f49-rs8jl teleport ERROR REPORT:
teleport-7f79f44f49-rs8jl teleport Original Error: *trace.BadParameterError acme can&#39;t get a cert for domain teleport.cluster.local, add it to the proxy_service.public_addr, or use one of the domains: teleport.XXX.XXX
teleport-7f79f44f49-rs8jl teleport Stack Trace:
teleport-7f79f44f49-rs8jl teleport      /go/src/github.com/gravitational/teleport/lib/service/acme.go:70 github.com/gravitational/teleport/lib/service.(*hostPolicyChecker).checkHost
teleport-7f79f44f49-rs8jl teleport      /go/src/github.com/gravitational/teleport/vendor/golang.org/x/crypto/acme/autocert/autocert.go:301 golang.org/x/crypto/acme/autocert.(*Manager).GetCertificate
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/common.go:1017 crypto/tls.(*Config).getCertificate
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/handshake_server_tls13.go:377 crypto/tls.(*serverHandshakeStateTLS13).pickCertificate
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/handshake_server_tls13.go:52 crypto/tls.(*serverHandshakeStateTLS13).handshake
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/handshake_server.go:51 crypto/tls.(*Conn).serverHandshake
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/crypto/tls/conn.go:1391 crypto/tls.(*Conn).Handshake
teleport-7f79f44f49-rs8jl teleport      /go/src/github.com/gravitational/teleport/lib/multiplexer/web.go:135 github.com/gravitational/teleport/lib/multiplexer.(*WebListener).detectAndForward
teleport-7f79f44f49-rs8jl teleport      /opt/go/src/runtime/asm_amd64.s:1371 runtime.goexit
teleport-7f79f44f49-rs8jl teleport User Message: acme can&#39;t get a cert for domain teleport.cluster.local, add it to the proxy_service.public_addr, or use one of the domains: teleport.XXX.XXX] multiplexer/web.go:137

gravitational / teleport