search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
16.96k stars 1.7k forks source link

tsh scp should work with create_host_user #22710

Open programmerq opened 1 year ago

programmerq commented 1 year ago

Expected behavior:

When using the create_host_user feature that dynamically adds/removes a host user on an ssh node, tsh scp should work as expected.

Current behavior:

% tsh ssh ubuntu@testscp
$ whoami
ubuntu
$ ^D
the connection was closed on the remote side on  07 Mar 23 09:34 CST

% tsh scp ./admin.yaml ubuntu@testscp:/tmp/admin.yaml
ERROR: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF

ERROR: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF

The ubuntu user that does not already exist on the host is successfully auto-created and auto-deleted when using tsh ssh but not when using tsh scp.

Bug details:

ssh_service debug logs on the testscp node:

2023-03-07T15:39:17Z DEBU [PROXY:AGE] Transport request: teleport-transport. leaseID:1 target:teleport.example.com:443 cluster:example reversetunnel/agent.go:570
2023-03-07T15:39:17Z DEBU [PROXY:AGE] Received out-of-band proxy transport request for @local-node [59cc22c7-ea7d-4c64-a3db-f18418ff97e1.other]. cluster:example reversetunnel/transport.go:199
2023-03-07T15:39:17Z DEBU [PROXY:AGE] Handing off connection to a local "node" service. cluster:example reversetunnel/transport.go:274
2023-03-07T15:39:18Z DEBU [NODE]      conn(12.30.31.196:54121->172.17.0.2:50072, user=ubuntu) auth attempt fingerprint:ssh-rsa-cert-v01@openssh.com SHA256:eX2oqflomxe35/B+cSNIZxVY+iuJpYJsEkHNN8DUGT4 local:172.17.0.2:50072 remote:12.30.31.196:54121 user:ubuntu srv/authhandlers.go:291
2023-03-07T15:39:18Z DEBU [NODE]      conn(12.30.31.196:54121->172.17.0.2:50072, user=ubuntu) auth attempt with key ssh-rsa-cert-v01@openssh.com SHA256:eX2oqflomxe35/B+cSNIZxVY+iuJpYJsEkHNN8DUGT4, &ssh.Certificate{Nonce:[]uint8{0x6e, 0xa3, 0x8, 0x8b, 0x73, 0x37, 0xbd, 0x6c, 0xe5, 0xb5, 0x39, 0x3, 0xee, 0xf3, 0x52, 0x3b, 0x20, 0x4, 0xb5, 0xfb, 0x40, 0x45, 0x7e, 0x9, 0x39, 0x5d, 0x74, 0xfe, 0x9, 0x10, 0xb0, 0x97}, Key:(*ssh.rsaPublicKey)(0xc00104d260), Serial:0x0, CertType:0x1, KeyId:"jeff", ValidPrincipals:[]string{"root", "ubuntu", "ec2-user", "admin", "jeff", "jefferya", "-teleport-internal-join"}, ValidAfter:0x640755cf, ValidBefore:0x6407fecb, Permissions:ssh.Permissions{CriticalOptions:map[string]string{}, Extensions:map[string]string{"login-ip":"12.30.31.196", "permit-X11-forwarding":"", "permit-agent-forwarding":"", "permit-port-forwarding":"", "permit-pty":"", "private-key-policy":"none", "teleport-roles":"{\"version\":\"v1\",\"roles\":[\"admin\",\"contractor\",\"auto-users\"]}", "teleport-route-to-cluster":"other", "teleport-traits":"{\"logins\":[\"root\",\"ubuntu\",\"ec2-user\",\"admin\",\"jeff\",\"jefferya\"]}"}}, Reserved:[]uint8{}, SignatureKey:(*ssh.rsaPublicKey)(0xc00104d2a0), Signature:(*ssh.Signature)(0xc000e02880)} fingerprint:ssh-rsa-cert-v01@openssh.com SHA256:eX2oqflomxe35/B+cSNIZxVY+iuJpYJsEkHNN8DUGT4 local:172.17.0.2:50072 remote:12.30.31.196:54121 user:ubuntu srv/authhandlers.go:294
2023-03-07T15:39:18Z DEBU [NODE]      Successfully authenticated fingerprint:ssh-rsa-cert-v01@openssh.com SHA256:eX2oqflomxe35/B+cSNIZxVY+iuJpYJsEkHNN8DUGT4 local:172.17.0.2:50072 remote:12.30.31.196:54121 user:ubuntu srv/authhandlers.go:381
2023-03-07T15:39:18Z DEBU [NODE]      Checking permissions for (jeff,ubuntu) to login to node with RBAC checks. srv/authhandlers.go:546
2023-03-07T15:39:18Z DEBU [RBAC]      Access to node "59cc22c7-ea7d-4c64-a3db-f18418ff97e1" granted, allow rule in role "admin" matched. services/role.go:2488
2023-03-07T15:39:18Z DEBU [SSH:NODE]  Incoming connection 12.30.31.196:54121 -> 172.17.0.2:50072 version: SSH-2.0-Go, certtype: "user" sshutils/server.go:502
2023-03-07T15:39:18Z DEBU [KEEPALIVE] Starting keep-alive loop with interval 5m0s and max count 3. srv/keepalive.go:64
2023-03-07T15:39:18Z DEBU [NODE]      Handling request subsystem, want reply true. id:13 local:172.17.0.2:50072 login:ubuntu remote:12.30.31.196:54121 teleportUser:jeff regular/sshserver.go:1533
2023-03-07T15:39:18Z DEBU [NODE]      Subsystem request: &{<nil> <nil> <nil> 0xc0004ec690}. id:13 local:172.17.0.2:50072 login:ubuntu remote:12.30.31.196:54121 teleportUser:jeff regular/sshserver.go:1771
2023-03-07T15:39:18Z DEBU [SUBSYSTEM] starting SFTP process regular/sftp.go:108
Failed to launch: user: unknown user ubuntu.
2023-03-07T15:39:18Z DEBU [SUBSYSTEM] SFTP process finished regular/sftp.go:187
2023-03-07T15:39:18Z WARN [SUBSYSTEM] Connection problem. error:[write |1: broken pipe] regular/sftp.go:193
2023-03-07T15:39:18Z DEBU             Subsystem &{0xc00166e2c0 {0xc000dc0480 0xc0007122c0} 0xc001048840 0xc0004ec690} finished with result: exit status 255, write |1: broken pipe. regular/sshserver.go:1781
2023-03-07T15:39:18Z DEBU [NODE]      Close session request: exit status 255, write |1: broken pipe. id:13 local:172.17.0.2:50072 login:ubuntu remote:12.30.31.196:54121 teleportUser:jeff regular/sshserver.go:1480
2023-03-07T15:39:18Z DEBU [NODE]      Releasing associated resources - context has been closed. id:13 local:172.17.0.2:50072 login:ubuntu remote:12.30.31.196:54121 teleportUser:jeff srv/monitor.go:266
2023-03-07T15:39:18Z INFO [AUDIT]     session.data addr.remote:12.30.31.196:54121 code:T2006I ei:2.147483646e+09 event:session.data login:ubuntu namespace:default rx:4238 server_id:59cc22c7-ea7d-4c64-a3db-f18418ff97e1 sid: time:2023-03-07T15:39:18.733Z tx:4834 uid:01f228bb-5647-4128-a438-4af4618b1b6d user:jeff events/emitter.go:265
2023-03-07T15:39:18Z DEBU [SSH:NODE]  Closed connection 12.30.31.196:54121. sshutils/server.go:507
Gunni commented 1 year ago

Workaround that worked for me:

  1. connect using tsh ssh
  2. While that session is connected, in a different terminal copy file using tsh scp
double-em commented 1 year ago

Workaround that worked for me:

  1. connect using tsh ssh
  2. While that session is connected, in a different terminal copy file using tsh scp

I can confirm this workaround worked for me too. Found the workaround accidentally when me and a co-worker was investigating the issue and I was able to transfer files and he was not. We discovered that we need an active user when having 'create_host_user' on.

My versions: Teleport v12.2.4 git:v12.2.4-0-g0f5a2d8 go1.20.3 Proxy version: 12.2.3

jakubbujny commented 1 year ago

I confirm it's still the case in teleport v13, causing problems while using Ansible

[WARNING]: sftp transfer mechanism failed on [teleport--test-3]. Use ANSIBLE_DEBUG=1 to see detailed information
[WARNING]: scp transfer mechanism failed on [teleport--test-3]. Use ANSIBLE_DEBUG=1 to see detailed information
[WARNING]: piped transfer mechanism failed on [teleport--test-3]. Use ANSIBLE_DEBUG=1 to see detailed information

fatal: [teleport--test-3]: FAILED! => 
  msg: |-
    failed to transfer file to /<redacted>/AnsiballZ_setup.py:

    dd: failed to open '/<redacted>/AnsiballZ_setup.py': No such file or directory
jakubbujny commented 1 year ago

Here is workaround which I was able to make based on previous comments:

tsh ssh "user@<IP>" "while true; do sleep 30; done" &
# save PID to kill process after work
pid=$!
# make echo first to avoid race condition with background process
tsh ssh "user@<IP>" "echo"
<do any stuff like e.g. Ansible playbook>
# kill background process
kill -9 "${pid}"
bediverus commented 7 months ago

I recently switched to teleport v14, tsh scp also does not work with users created by Teleport.

tunguyen9889 commented 4 months ago

Any update for this issue, please? We're using Teleport v15 and facing the same problem, too.