`tsh config` generates invalid config for proxied mode

[FireDrunk]

Expected behavior: tsh config generates a configuration that uses port 443

Current behavior: tsh config generates a configuration that uses port 3022

Bug details:

• v12.1.1
• Run cluster in Proxy mode (only expose port 443), generate a config using tsh config, and try to use it.

The currently generated tsh config output:

# Begin generated Teleport configuration for *obfuscated* by tsh

# Common flags for all teleport hosts
Host *.teleport *obfuscated*
    UserKnownHostsFile "/home/thijs/.tsh/known_hosts"
    IdentityFile "/home/thijs/.tsh/keys/*obfuscated*/teleport-admin"
    CertificateFile "/home/thijs/.tsh/keys/*obfuscated*/teleport-admin-ssh/teleport-cert.pub"
    HostKeyAlgorithms rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com

# Flags for all teleport hosts except the proxy
Host *.teleport !*obfuscated*
    Port 3022
    ProxyCommand "/usr/local/bin/tsh" proxy ssh --cluster=teleport --proxy=*obfuscated* %r@%h:%p

# End generated Teleport configuration

This contains port 3022, which is invalid. I've tried setting this to 443 but that doesn't fix the issue.

The logs (when changing Port 3022 -> Port 443 and adding --debug to the tsh command.

ssh -vvv -F ./ssh_config_teleport *obfuscated*@*obfuscated*.teleport
OpenSSH_8.8p1, OpenSSL 3.0.8 7 Feb 2023
debug1: Reading configuration data ./ssh_config_teleport
debug1: ./ssh_config_teleport line 4: Applying options for *.teleport
debug1: ./ssh_config_teleport line 11: Applying options for *.teleport
debug1: Executing proxy command: exec "/usr/local/bin/tsh" --debug proxy ssh --cluster=teleport --proxy=*obfuscated* *obfuscated*@*obfuscated*.teleport:443
debug1: identity file /home/thijs/.tsh/keys/*obfuscated*/teleport-admin type 0
debug1: certificate file /home/thijs/.tsh/keys/*obfuscated*/teleport-admin-ssh/teleport-cert.pub type 4
debug1: Local version string SSH-2.0-OpenSSH_8.8
2023-03-28T11:23:05+02:00 INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/tmp/ssh-XXXXXXxThEWc/agent.3843" client/api.go:3889
2023-03-28T11:23:05+02:00 DEBU [KEYSTORE]  Reading certificates from path "/home/thijs/.tsh/keys/*obfuscated*/teleport-admin-ssh/teleport-cert.pub". client/keystore.go:339
2023-03-28T11:23:05+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-03-28 19:41:47 +0000 UTC". client/client_store.go:91
2023-03-28T11:23:05+02:00 INFO [KEYAGENT]  Loading SSH key for user "teleport-admin" and cluster "teleport". client/keyagent.go:195
2023-03-28T11:23:05+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-03-28 19:41:47 +0000 UTC". client/client_store.go:91
2023-03-28T11:23:05+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-03-28 19:41:47 +0000 UTC". client/client_store.go:91
2023-03-28T11:23:05+02:00 DEBU [KEYAGENT]  "Checking key: *obfuscated* client/keyagent.go:367
2023-03-28T11:23:05+02:00 DEBU [KEYAGENT]  Validated host *obfuscated*:443. client/keyagent.go:373
2023-03-28T11:23:05+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-03-28 19:41:47 +0000 UTC". client/client_store.go:91

ERROR REPORT:
Original Error: *errors.errorString ssh: subsystem request failed
Stack Trace:
    github.com/gravitational/teleport/tool/tsh/proxy.go:220 main.sshProxy
    github.com/gravitational/teleport/tool/tsh/proxy.go:79 main.onProxyCommandSSH.func1
    github.com/gravitational/teleport/lib/client/api.go:504 github.com/gravitational/teleport/lib/client.RetryWithRelogin
    github.com/gravitational/teleport/tool/tsh/proxy.go:67 main.onProxyCommandSSH
    github.com/gravitational/teleport/tool/tsh/tsh.go:1145 main.Run
    github.com/gravitational/teleport/tool/tsh/tsh.go:488 main.main
    runtime/proc.go:250 runtime.main
    runtime/asm_amd64.s:1594 runtime.goexit
User Message: ssh: subsystem request failed
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

When using tsh ssh <user>@<host> everything works fine

[strideynet]

The 3022 in this config actually refers to the port that it should try to connect to on the node on the other side of the proxy, rather than the port on the proxy. This default works where individuals are using the default SSH node ports. Is it possible in your situation that your nodes are using a different port ?

[FireDrunk]

@strideynet No everything on the client/node side is default.

sudo ss -tulpen | grep -i teleport
tcp   LISTEN 0      128                *:3022             *:*    users:(("teleport",pid=1212552,fd=27)) ino:12206299 sk:1003 v6only:0 <->     
tcp   LISTEN 0      128                *:3025             *:*    users:(("teleport",pid=1212552,fd=14)) ino:12206035 sk:1004 v6only:0 <->     
tcp   LISTEN 0      128                *:443              *:*    users:(("teleport",pid=1212552,fd=21)) ino:12206293 sk:1006 v6only:0 <-> 

The port is also properly responding:

[root@vm-sandbox-shared-teleport-001 ~]# nc -v localhost 3022
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to ::1:3022.
SSH-2.0-Teleport

My teleport.yaml

---
version: v3
teleport:
  nodename: vm-sandbox-shared-teleport-001
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: ""
  diag_addr: ""

auth_service:
  enabled: "yes"
  tokens:
    - "node:TEST_TOKEN"
  listen_addr: 0.0.0.0:3025
  cluster_name: teleport
  proxy_listener_mode: multiplex
  authentication:
    second_factor: on
    webauthn:
      rp_id: *external_url*

ssh_service:
  enabled: "yes"
  commands:
    - name: hostname
      command: [hostname]
      period: 1m0s

proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:443
  public_addr: *external_url*:443
  https_keypairs:
    - cert_file: /etc/pki/tls/certs/teleport.crt
      key_file: /etc/pki/tls/private/teleport.key
  https_keypairs_reload_interval: 0s

(Yes, we are aware that the static token is insecure ;) )

[FireDrunk]

Some more debugging:

tsh proxy ssh --cluster=teleport --proxy=<proxy>:443 <user>@<node>:22
SSH-2.0-Teleport

It seems like the proxy comes online properly...

[strideynet]

If you add -d to the tsh in the ssh_config, and retain the 3022 port as originally generated. What do you see in your logs ? Do you see any interesting logs in your proxy around this time as well ?

[strideynet]

Oh - and by any chance - does your hostname you are connecting to contain capital letters ?

[FireDrunk]

@strideynet Ah yes, indeed it does! After checking the logs I'm seeing DNS errors on resolving the hostname without capitals.

Mar 29 07:16:11 vm-sandbox-shared-teleport-001 teleport[1212552]: 2023-03-29T07:16:11Z [PROXY]     WARN "Subsystem request proxySubsys(cluster=default/teleport, host=*obfuscated*, port=3022) failed: Teleport proxy failed to connect to \"node\" agent \"*obfuscated*:3022\" over direct dial:\n\n  dial tcp: lookup *obfuscated* on 168.63.129.16:53: no such host\n\nThis usually means that the agent is offline or has disconnected. Check the\nagent logs and, if the issue persists, try restarting it or re-registering it\nwith the cluster.." id:76 local:*obfuscated*:443 login:*obfuscated* remote:*obfuscated*:9396 teleportUser:teleport-admin regular/sshserver.go:1780

Seems that it's trying to find the node directly instead of using the tunnel.

[strideynet]

So unfortunately, you are running into a bit of a long-standing bug with hostnames involving capital letters and openssh https://github.com/gravitational/teleport/issues/16457 / https://github.com/gravitational/teleport/issues/2833

Depending on your openssh version (it must be newer than 8.1p0), you may be able to swap out the %h in the generated config for %n.

[FireDrunk]

@strideynet After changing the %h to %n I'm still refused because no valid key has been found. My client is using OpenSSH 8.8p1, the server has OpenSSH version is 8.0p1-17.el8_7

[zmb3]

@stevenGravy @strideynet is this still an issue after #27536?

[stevenGravy]

This should be good @zmb3 .