Improve teleport-kube-agent error message when using the wrong proxy port

[fraenkel]

Description

Last week the chart was working fine. But this week any new deployment fails.

What happened:

The log contains

ERRO [PROC:1]    Critical service kube.init has exited with error set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out, aborting. service/signals.go:145
error: set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out

What you expected to happen: Success

How to reproduce it (as minimally and precisely as possible):

Environment

• Teleport version (use teleport version): 5.1.0

• OS (e.g. from /etc/os-release): EKS

• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS

Relevant Debug Logs If Applicable

DEBU [SQLITE]    Connected to: file:/var/lib/teleport/proc/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:173
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:218
INFO [PROC]      Adding diagnostic debugging handlers. To connect with profiler, use `go tool pprof 0.0.0.0:3000`. service/service.go:1948
DEBU [PROC]      Adding service to supervisor. service:readyz.monitor service/supervisor.go:181
INFO [PROC:1]    Service diag is creating new listener on 0.0.0.0:3000. service/signals.go:214
INFO [DIAG:1]    Starting diagnostic service on 0.0.0.0:3000. service/service.go:2020
DEBU [PROC]      Adding service to supervisor. service:diagnostic.service service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:diagnostic.shutdown service/supervisor.go:181
DEBU [KEYGEN]    SSH cert authority started with no keys pre-compute. native/native.go:107
DEBU [PROC]      Adding service to supervisor. service:register.kube service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:kube.init service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:common.rotate service/supervisor.go:181
DEBU [PROC:1]    Service has started. service:readyz.monitor service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:register.kube service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:diagnostic.service service/supervisor.go:242
DEBU [PROC:1]    No signal pipe to import, must be first Teleport process. service/service.go:761
DEBU [PROC:1]    Service has started. service:kube.init service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:common.rotate service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:diagnostic.shutdown service/supervisor.go:242
DEBU [PROC:1]    Connected state: never updated. service/connect.go:99
INFO [PROC]      Connecting to the cluster xxx.yyy.zzz with TLS client certificate. service/connect.go:127
DEBU [PROC]      Attempting to connect to Auth Server directly. service/connect.go:793
DEBU [PROC]      Connected to Auth Server with direct connection. service/connect.go:811
DEBU [PROC:1]    Connected client: Identity(Kube, cert(9b664e5d-f6bc-4ee0-b152-2c8c7f9b19a8.xxx.yyy.zzz issued by xxx.yyy.zzz:130273960671349564572426993159963784294),trust root(xxx.yyy.zzz:130273960671349564572426993159963784294)) service/connect.go:81
DEBU [PROC:1]    Connected server: Identity(Kube, cert(9b664e5d-f6bc-4ee0-b152-2c8c7f9b19a8.xxx.yyy.zzz issued by xxx.yyy.zzz:130273960671349564572426993159963784294),trust root(xxx.yyy.zzz:130273960671349564572426993159963784294)) service/connect.go:82
DEBU [PROC]      Adding service to supervisor. service:auth.client.kube service/supervisor.go:181
DEBU [PROC:1]    Broadcasting event. event:KubeIdentity service/supervisor.go:332
DEBU [PROC:1]    Service is completed and removed. service:register.kube service/supervisor.go:219
DEBU [KUBERNETE] Received event "KubeIdentity". service/kubernetes.go:55
DEBU [PROC:1]    Service has started. service:auth.client.kube service/supervisor.go:242
DEBU [PROC:1]    Creating sqlite backend for [kubernetes]. service/service.go:1443
DEBU [SQLITE]    Connected to: file:/var/lib/teleport/cache/kubernetes/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 100ms lite/lite.go:173
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:218
DEBU [AUTH]      GRPC(CLIENT): keep alive 1m0s count: 3. auth/clt.go:320
INFO [KUBERNETE] Cache "kube" first init succeeded. cache/cache.go:574
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:1867
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:1867
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions. service/service.go:1867
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions/default. service/service.go:1867
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:1867
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:1867
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming. service/service.go:1867
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming/default. service/service.go:1867
DEBU [PROC]      Adding service to supervisor. service:uploader.service service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:uploader.shutdown service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:fileuploader.service service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:fileuploader.shutdown service/supervisor.go:181
DEBU [PROC:1]    Service has started. service:uploader.service service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:uploader.shutdown service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:fileuploader.service service/supervisor.go:242
WARN [PROC:1]    Teleport process has exited with error: set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out service:kube.init service/supervisor.go:247
DEBU [PROC:1]    Broadcasting event. event:ServiceExitedWithError service/supervisor.go:332
DEBU [PROC:1]    Service has started. service:fileuploader.shutdown service/supervisor.go:242
DEBU [PROC:1]    Service is completed and removed. service:kube.init service/supervisor.go:219
WARN [KUBERNETE] Re-init the cache on error: grpc: the client connection is closing. cache/cache.go:627
DEBU [KUBERNETE] Reloading Linear(attempt=0, duration=0s). cache/cache.go:635
ERRO [PROC:1]    Critical service kube.init has exited with error set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out, aborting. service/signals.go:145
WARN [KUBERNETE] Re-init the cache on error: grpc: the client connection is closing. cache/cache.go:627
DEBU [KUBERNETE] Reloading Linear(attempt=1, duration=1s). cache/cache.go:635
DEBU [KUBERNETE] Cache is closing, returning from update loop. cache/cache.go:613
DEBU [PROC:1]    Broadcasting event. event:TeleportExit service/supervisor.go:332

ERROR REPORT:
Original Error: *trace.BadParameterError set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out
Stack Trace:
    /go/src/github.com/gravitational/teleport/lib/service/kubernetes.go:113 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initKubernetesService
    /go/src/github.com/gravitational/teleport/lib/service/kubernetes.go:60 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initKubernetes.func1
    /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:450 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
    /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:242 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
    /opt/go/src/runtime/asm_amd64.s:1375 runtime.goexit
User Message: set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out

[webvictim]

Could you please share the full Helm command that you used to install the chart?

[fraenkel]

helm install teleport-kube-agent . --create-namespace --namespace teleport --set proxyAddr=xxx.yyy.zzz --set authToken=akubetoken --set kubeClusterName=testcluster

[webvictim]

Does your proxyAddr contain a port? It should have :3080 on the end to point to a proxy, or maybe :443 if you're running behind a load balancer.

[fraenkel]

Adding 443 fixed it... guess I did it and forgot that I did.

[webvictim]

@awly Maybe we should consider making the proxy part of this error more obvious?

[awly]

Yeah, the message should not hide the underlying error (connection refused) and should suggest checking the port in auth_servers