search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.68k stars 1.77k forks source link

set kubernetes_service.listen_addr if this process can be reached: Teleport Kube Agent #11813

Closed benarent closed 2 years ago

benarent commented 2 years ago

Description

What happened: Our Instruqt Kubernetes demo is broken as the Helm Kube agents isn't able to connection. This ticket has the teleport.yaml and debug log associated with it.

teleport.yaml

teleport:
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
auth_service:
  enabled: "yes"
  cluster_name: teleportvm
  listen_addr: 0.0.0.0:3025
  proxy_listener_mode: multiplex
  authentication:
    type: local
    second_factor: optional
    u2f:
      app_id: https://$HOSTNAME.${INSTRUQT_PARTICIPANT_ID}.instruqt.io:443
      facets:
        - https://$HOSTNAME.${INSTRUQT_PARTICIPANT_ID}.instruqt.io:443
        - https://$HOSTNAME.${INSTRUQT_PARTICIPANT_ID}.instruqt.io
        - $HOSTNAME.${INSTRUQT_PARTICIPANT_ID}.instruqt.io:443
        - $HOSTNAME.${INSTRUQT_PARTICIPANT_ID}.instruqt.io
    webauthn:
      rp_id: "$HOSTNAME.${INSTRUQT_PARTICIPANT_ID}.instruqt.io"
  tokens:
    - "kube,node,db,app:TOKEN"
ssh_service:
  enabled: "yes"
  labels:
    env: example
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
app_service:
  enabled: "yes"
  debug_app: true
  apps:
  - name: "k8s-dashboard"
    uri: "http://kubernetes-vm:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/"
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:443
  kube_listen_addr: 0.0.0.0:3026 
  public_addr: $HOSTNAME.${INSTRUQT_PARTICIPANT_ID}.instruqt.io:443
  acme:
    enabled: yes
    email: "ben@goteleport.com"

Adding Helm Chart

PROXY_ADDR=teleportvm.${INSTRUQT_PARTICIPANT_ID}.[instruqt.io](http://instruqt.io/)
PROXY_ADDR=teleportvm.dkl8bje9lxmn.instruqt.io
CLUSTER='cookie'
TOKEN='TOKEN'
helm install teleport-agent teleport/teleport-kube-agent --set kubeClusterName=${CLUSTER?} \
  --set proxyAddr=${PROXY_ADDR?} --set authToken=${TOKEN?} --create-namespace --namespace=teleport-agent
Debug Log from Kubernetes
root@kubernetes-vm:~# kubectl logs teleport-agent-7f8bbd78c8-cwnlm  --namespace=teleport-agent
2022-04-07T21:09:51Z INFO [PROC:1]    Service diag is creating new listener on 0.0.0.0:3000. service/signals.go:213
2022-04-07T21:09:51Z INFO [DIAG:1]    Starting diagnostic service on 0.0.0.0:3000. service/service.go:2317
2022-04-07T21:09:51Z INFO [PROC:1]    Connecting to the cluster teleportvm with TLS client certificate. service/connect.go:160
2022-04-07T21:09:51Z INFO [PROC:1]    Kube: features loaded from auth server: Kubernetes:true App:true DB:true Desktop:true  service/connect.go:72
2022-04-07T21:09:51Z INFO [KUBERNETE] Cache "kube" first init succeeded. cache/cache.go:713
2022-04-07T21:09:51Z INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:2099
2022-04-07T21:09:51Z INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:2099
2022-04-07T21:09:51Z INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions. service/service.go:2099
2022-04-07T21:09:51Z INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions/default. service/service.go:2099
2022-04-07T21:09:51Z INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:2099
2022-04-07T21:09:51Z INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:2099
2022-04-07T21:09:51Z INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming. service/service.go:2099
2022-04-07T21:09:51Z INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming/default. service/service.go:2099
2022-04-07T21:09:51Z WARN [PROC:1]    Teleport process has exited with error. error:[
ERROR REPORT:
Original Error: *trace.BadParameterError set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out
Stack Trace:
        /go/src/github.com/gravitational/teleport/lib/service/kubernetes.go:115 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initKubernetesService
        /go/src/github.com/gravitational/teleport/lib/service/kubernetes.go:62 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initKubernetes.func1
        /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:494 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:263 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out] service:kube.init service/supervisor.go:268
2022-04-07T21:09:51Z ERRO [PROC:1]    Critical service kube.init has exited with error set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out, aborting. service/signals.go:144
ERROR: set kubernetes_service.listen_addr if this process can be reached from a teleport proxy or point teleport.auth_servers to a proxy to dial out

Version: Teleport 9.0.3

webvictim commented 2 years ago

This is the config that the teleport-cluster Helm chart uses when proxyListenerMode: multiplex is set:

version: v2
teleport:
  log:
    severity: INFO
    output: stderr
    format:
      output: text
      extra_fields: ["timestamp","level","component","caller"]
auth_service:
  enabled: true
  cluster_name: teleport.example.com
  authentication:
    type: local
    second_factor: on
    webauthn:
      rp_id: teleport.example.com
  proxy_listener_mode: multiplex
kubernetes_service:
  enabled: true
  listen_addr: 0.0.0.0:443
  kube_cluster_name: teleport.example.com
proxy_service:
  public_addr: 'teleport.example.com:443'
  enabled: true
ssh_service:
  enabled: false

Here's the matching teleport-kube-agent config:

teleport:
  auth_token: "token-goes-here"
  auth_servers: ["teleport.example.com:443"]
  log:
    severity: INFO
    output: stderr
    format:
      output: text
      extra_fields: ["timestamp","level","component","caller"]

kubernetes_service:
  enabled: true
  kube_cluster_name: test-kube-cluster

app_service:
  enabled: false
db_service:
  enabled: false
auth_service:
  enabled: false
ssh_service:
  enabled: false
proxy_service:
  enabled: false
benarent commented 2 years ago

I'm not sure what fixed this, but I made a few changes.

  1. I was missing port from my poxy PROXY_ADDR=teleportvm.dkl8bje9lxmn.instruqt.io -> PROXY_ADDR=teleportvm.dkl8bje9lxmn.instruqt.io:443
    1. I added teleport version: v2 to my Teleport.yaml for multiplexing.