search

microsoft / vscode-remote-release

Visual Studio Code Remote Development: Open any folder in WSL, in a Docker container, or on a remote machine using SSH and take advantage of VS Code's full feature set.
https://aka.ms/vscode-remote
Other
3.67k stars 291 forks source link

Failed to use local server mode with tsh #6469

Closed KasperSkytte closed 1 year ago

KasperSkytte commented 2 years ago

Description

I am having some trouble connecting to remote hosts through vscode. My setup might be unusual, but it ought to work. I use a https://goteleport.com bastion host to connect to my different servers, where some are on the same local network, some are behind NAT routers on different institutional networks. I can connect just fine using the local terminal both using the dedicated tsh client as well as the standard Linux OpenSSH client by setting some proxy options in ~/.ssh/config as described here. This does not seem to be related to Teleport itself as it's simply acting as a SSH proxy, however, hence I post here.

Judging from the logs, the connection is established and user is authenticated correctly and VSCode does succesfully run several commands on the remote, but fails with Failed to parse remote port from server output. To me it seems like a BASH script that doesn't parse variables correctly, but I have no way of knowing for sure without inspecting it. I have tried setting "remote.SSH.enableDynamicForwarding": false with no difference. Setting "remote.SSH.remoteServerListenOnSocket": false results in a limited connection, where I cannot start a terminal, open a folder, or do anything really, and two popup messages show could not fetch remote environment and Failed to connect to the remote extension host server (Error: WebSocket close with status code 1006). I make sure to kill the code-server on both host and remote before connecting after adjusting these settings.

As mentioned, I can connect succesfully using the local terminal separately from VSCode using both ssh and tsh.

Steps to Reproduce:

This is probably not easy to reproduce, and would not necessarily involve teleport. Any other way of setting up a SSH proxy would presumably have the same issue. But possibly:

System details and logs

SSH config file
``` # # Begin generated Teleport configuration for teleport.domain.com:443 from `tsh config` # # Common flags for all teleport.domain.com hosts Host *.teleport.domain.com teleport.domain.com UserKnownHostsFile /home/user/.tsh/known_hosts IdentityFile "/home/user/.tsh/keys/teleport.domain.com/user" CertificateFile "/home/user/.tsh/keys/teleport.domain.com/user-ssh/teleport.domain.com-cert.pub" # Flags for all teleport.domain.com hosts except the proxy Host *.teleport.domain.com !teleport.domain.com Port 3022 ProxyCommand /usr/local/bin/tsh proxy ssh --cluster=teleport.domain.com --proxy=teleport.domain.com %r@%h:%p # End generated Teleport configuration Host server.teleport.domain.com HostName server.teleport.domain.com ```
Log
``` [10:34:45.907] Log Level: 2 [10:34:45.908] remote-ssh@0.77.2022031115 [10:34:45.908] linux x64 [10:34:45.911] SSH Resolver called for "ssh-remote+server.teleport.domain.com", attempt 1 [10:34:45.911] "remote.SSH.useLocalServer": true [10:34:45.911] "remote.SSH.path": undefined [10:34:45.911] "remote.SSH.configFile": undefined [10:34:45.911] "remote.SSH.useFlock": true [10:34:45.911] "remote.SSH.lockfilesInTmp": false [10:34:45.912] "remote.SSH.localServerDownload": auto [10:34:45.912] "remote.SSH.remoteServerListenOnSocket": false [10:34:45.913] "remote.SSH.showLoginTerminal": false [10:34:45.913] "remote.SSH.defaultExtensions": [] [10:34:45.914] "remote.SSH.loglevel": 2 [10:34:45.914] "remote.SSH.enableDynamicForwarding": true [10:34:45.914] "remote.SSH.enableRemoteCommand": true [10:34:45.914] "remote.SSH.serverPickPortsFromRange": {} [10:34:45.914] "remote.SSH.serverInstallPath": {} [10:34:45.925] SSH Resolver called for host: server.teleport.domain.com [10:34:45.925] Setting up SSH remote "server.teleport.domain.com" [10:34:45.928] Acquiring local install lock: /tmp/vscode-remote-ssh-0728281a-install.lock [10:34:45.931] Looking for existing server data file at /home/user/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0728281a-c722ca6c7eed3d7987c0d5c3df5c45f6b15e77d1-0.77.2022031115/data.json [10:34:45.932] Using commit id "c722ca6c7eed3d7987c0d5c3df5c45f6b15e77d1" and quality "stable" for server [10:34:45.936] Install and start server if needed [10:34:45.944] PATH: /home/user/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin [10:34:45.944] Checking ssh with "ssh -V" [10:34:45.951] > OpenSSH_8.4p1 Ubuntu-5ubuntu1.2, OpenSSL 1.1.1j 16 Feb 2021 [10:34:45.964] askpass server listening on /run/user/1000/vscode-ssh-askpass-9b3bbf1af18957a6c490374ed24893e9c9f950e3.sock [10:34:45.964] Spawning local server with {"serverId":1,"ipcHandlePath":"/run/user/1000/vscode-ssh-askpass-a664bcf65c2c0a14a1cb561a2188bada21eb1652.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","40225","-o","ConnectTimeout=15","server.teleport.domain.com"],"serverDataFolderName":".vscode-server","dataFilePath":"/home/user/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0728281a-c722ca6c7eed3d7987c0d5c3df5c45f6b15e77d1-0.77.2022031115/data.json"} [10:34:45.964] Local server env: {"SSH_AUTH_SOCK":"/run/user/1000/keyring/ssh","SHELL":"/bin/bash","DISPLAY":":1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/usr/share/code/code","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/run/user/1000/vscode-ssh-askpass-9b3bbf1af18957a6c490374ed24893e9c9f950e3.sock"} [10:34:45.968] Spawned 1310820 [10:34:46.041] > local-server-1> Spawned ssh, pid=1310829 [10:34:46.042] stderr> OpenSSH_8.4p1 Ubuntu-5ubuntu1.2, OpenSSL 1.1.1j 16 Feb 2021 [10:34:46.468] stderr> Authenticated to server.teleport.domain.com (via proxy). [10:34:46.536] > echo 'ready: 60f4becc7a6b' [10:34:46.562] > uname -rsv [10:34:46.616] > user@server:~$ echo 'ready: 60f4becc7a6b' > ready: 60f4becc7a6b > > user@server:~$ uname -rsv [10:34:46.621] > Linux 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022 [10:34:46.622] Platform: linux [10:34:46.622] > > user@server:~$ [10:34:46.622] Shell: user@server:~$ [10:34:46.622] Creating bash subshell inside " user@server:~$ " [10:34:46.647] > echo $SHELL > /bin/bash > > user@server:~$ bash [10:34:46.660] > user@server:~$ echo "bash version: " $BASH_VERSION > bash version: 5.0.17(1)-release > > user@server:~$ [10:34:46.707] > function mainScriptWrapper { > > > > echo "60f4becc7a6b: running" > > > o> # > > # Setup > > # > 1 > > VSC_TMP="${XDG_RUNTIME_DIR:-`echo /tmp`}" > > COMMIT_ID="c722ca6c7eed3d7987c0d5c3df5c45f6b15e77d1" > > SERVER_APP_NAME="code-server" > > EXTENSIONS="" > > TELEMETRY="--telemetry-level all" > > export VSCODE_AGENT_FOLDER=$HOME/.vscode-server > > ALLOW_CLIENT_DOWNLOAD="1" [10:34:46.708] > > [10:34:46.731] > FORCE_CLIENT_DOWNLOAD="0" > > now=`date +%s` > > > > PORT_FLAG=--port=0 > > > > SERVER_LISTEN_FLAG= > > UNPACK_RESULT= > > DID_LOCAL_DOWNLOAD=0 > > DOWNLOAD_TIME= > > INSTALL_TIME= > > EXT_INSTALL_TIME= > > SERVER_START_TIME= > > CONNECTION_TOKEN= > > > f> VSCH_DIR="$VSCODE_AGENT_FOLDER/bin/$COMMIT_ID" > > VSCH_SERVER_SCRIPT="$VSCH_DIR/bin/$SERVER_APP_NAME" > > VSCH_LOGFILE="$VSCODE_AGENT_FOLDER/.$COMMIT_ID.log" > > VSCH_PIDFILE="$VSCODE_AGENT_FOLDER/.$COMMIT_ID.pid" > > VSCH_TOKENFILE="$VSCODE_AGENT_FOLDER/.$COMMIT_ID.token" > > > > fail_with_exitcode() { > > [10:34:46.748] >  [10:34:46.748] > echo "60f4becc7a6b: start" > > [10:34:46.780] > echo "exitCode==$1==" > > [10:34:46.814] > echo_common_results > > [10:34:46.848] > echo "60f4becc7a6b: end" > > [10:34:46.849] Received install output: " > echo "exitCode==$1==" > echo_common_results > echo " [10:34:46.851] Failed to parse remote port from server output [10:34:46.851] Terminating local server [10:34:46.855] Resolver error: Error: at Function.Create (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:586041) at Object.t.handleInstallOutput (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:584693) at Object.e [as tryInstallWithLocalServer] (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:625033) at processTicksAndRejections (internal/process/task_queues.js:93:5) at async /home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:644070 at async Object.t.withShowDetailsEvent (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:647817) at async /home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:623505 at async b (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:620011) at async Object.t.resolveWithLocalServer (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:623120) at async Object.t.resolve (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:645427) at async /home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.77.2022031115/out/extension.js:1:722867 [10:34:46.871] ------ [10:34:46.873] Local server exit: 15 ```

Does this issue occur when you try this locally?: Yes Does this issue occur when you try this locally and all extensions are disabled?: Yes

KasperSkytte commented 2 years ago

In addition: I can connect just fine from behind NAT when the target is the machine running the proxy server docker container. Everything else no

tanhakabir commented 2 years ago

What is your default shell on your Remote? 

> 
> user@server:~$ 
[10:34:46.622] Shell: 
user@server:~$

Seems odd

KasperSkytte commented 2 years ago

@tanhakabir it's bash. I saw another issue https://github.com/microsoft/vscode-remote-release/issues/5111 where sh was to blame and changing to bash solved it. Not in my case, it's already bash. But notice the proxy command in ~/.ssh/config is /usr/local/bin/tsh, would that matter?

KasperSkytte commented 2 years ago

Adding "remote.SSH.useLocalServer": false to my VSCode settings seems to have solved the issue, see https://github.com/gravitational/teleport/issues/4176. But ideally it would be nice if the extension supported tsh too, or would that be up to teleport devs? There are compromises with using OpenSSH instead, no auditing or session recordings etc

tanhakabir commented 2 years ago

But notice the proxy command in ~/.ssh/config is /usr/local/bin/tsh, would that matter?

Are you able to use bash here instead and see what happens?

KasperSkytte commented 2 years ago

That would cause trouble. tsh is the standalone binary replacing ssh

webvictim commented 2 years ago

But notice the proxy command in ~/.ssh/config is /usr/local/bin/tsh, would that matter?

The ProxyCommand shouldn't make too much difference; tsh proxy ssh just calls ssh -s proxy:%h:%p with the identity/certificate files and some extra arguments underneath.

e.g. "C:\Users\Gus\tsh.exe" proxy ssh --cluster=teleport.example.com --proxy=teleport.example.com gus@apollo:3022 actually runs C:\Windows\System32\OpenSSH\ssh.exe -l gus -A -o UserKnownHostsFile=C:\Users\Gus\.tsh\known_hosts -p 3023 teleport.example.com -s proxy:apollo:3022@teleport.example.com

tanhakabir commented 2 years ago

Hm I haven't heard of tsh before so I don't know much about it. But in your original logs it's really odd to me that the install script that we pass to the remote was being just echo'd into the shell.

@webvictim it seems like using useLocalServer: false is a common workaround with tsh. Do you have any ideas how tsh changes the behavior and why we're seeing the script just being echo's and not executed with local server mode.

Quick TLDR of local server mode: we spin up a process to that performs the ssh connection to the remote. And then we pipe into this connected process the setup installation script to be executed on the remote to set up the VS Code server to connect to. In contrast the non local server pipes in the installations script directly into the ssh connection command. Something like echo installation script | ssh remote host.

KasperSkytte commented 2 years ago

I may be guessing here, but could it be due to the proxy command is invoking in a subshell (-s)?

KasperSkytte commented 2 years ago

Update: I tried replacing the ssh binary with tsh ssh by setting

"remote.SSH.enableRemoteCommand": true
"remote.SSH.path": "/usr/local/bin/tsh ssh"

But it errors and defaults to ssh

...
[13:01:15.206] Install and start server if needed
[13:01:16.598] Checking ssh with "/usr/local/bin/tsh ssh -V"
[13:01:16.611] Got error from ssh: spawn /usr/local/bin/tsh ssh ENOENT
[13:01:16.614] The specified path /usr/local/bin/tsh ssh is not a valid SSH binary
[13:04:44.749] PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
[13:04:44.749] Checking ssh with "ssh -V"
[13:04:44.763] > OpenSSH_8.4p1 Ubuntu-5ubuntu1.2, OpenSSL 1.1.1j  16 Feb 2021
...
tanhakabir commented 2 years ago

Is this a valid path to ssh? /usr/local/bin/tsh ssh Maybe you you need to escape the space?

webvictim commented 2 years ago

I may be guessing here, but could it be due to the proxy command is invoking in a subshell (-s)?

-s actually requests an SSH subsystem from the connected host - the proxy subsystem in this case which just opens a connection to the target node and passes stdin/stdout back to the caller.

Update: I tried replacing the ssh binary with tsh ssh

I think the issue is that VSCode is expecting an SSH version response from tsh ssh (which isn't implemented):

» tsh ssh -V
ERROR: unknown short flag '-V'

» ssh -V
OpenSSH_8.1p1, LibreSSL 2.7.3

@tanhakabir I don't know if there's a way to disable the SSH version check with VS Code?

@KasperSkytte If not, it'll probably need to be an ssh binary used instead of tsh ssh.

Hm I haven't heard of tsh before so I don't know much about it. But in your original logs it's really odd to me that the install script that we pass to the remote was being just echo'd into the shell.

For some context, tsh is the user-facing part of Teleport which (among other things) provides authentication and logging for SSH connections using short-lived certificates.

@webvictim it seems like using useLocalServer: false is a common workaround with tsh. Do you have any ideas how tsh changes the behavior and why we're seeing the script just being echo's and not executed with local server mode.

tsh ssh provides SSH-like connectivity and is broadly compatible with ssh, but doesn't guarantee to support all of its functionality or options.

Your explanation regarding local vs remote server is interesting and I'm not sure exactly why that doesn't work. One thing I think I've noticed is that when the server is already running on the remote end, you can connect to it using either local or remote server mode with tsh (as my understanding is that if there's a server process already running, it uses that rather than spawning a new one?). It just seems to be the initial setup/install process that doesn't work in local server mode. It seems likely that there's a slight technical implementation detail which works differently in terms of piping the input. I'd be curious to learn more about exactly how this works (link to code?) as it's likely something we can fix at the tsh level to make VS Code easier to use with Teleport :)

tanhakabir commented 2 years ago

I don't know if there's a way to disable the SSH version check with VS Code?

This would require an additional setting. It would be best if tsh could support -V since we already have a long list of settings and each additional setting introduces more branches for error.

Unfortunately I can't share the source code of the extension with you since it's closed source but generally the extension is expecting ssh behavior so there might be issues here with discrepancies.

A simple version of what we're doing is using node to spawn a process. In that process we run ssh remote-host and then we write into the process after connection some scripts. Do you have any ideas of the discrepancies between tsh and ssh that would be concerning with this flow?

KasperSkytte commented 2 years ago

Is this a valid path to ssh? /usr/local/bin/tsh ssh Maybe you you need to escape the space?

@tanhakabir Just tried escaping the space. Makes no difference, the setting is quoted so it parses the space correctly. It errors again at the version check, which invalidates tsh ssh as a binary because it doesn't return the same output as ssh -V. So I guess that's where something needs to be figured out

Just tried adding a fresh Ubuntu VM to the teleport bastion and then connecting through VSCode remote. It errors the first time, and then I set "remote.SSH.useLocalServer": false and it connects. Now I can set "remote.SSH.useLocalServer" back to true again and connect just fine. So after VSCode remote has done its thing at the first connection with a new host by running the scripts, everything's working as intended from there on

webvictim commented 2 years ago

@KasperSkytte That’s what I suspected too. Curious if when you reboot the VM (thus stopping the remote VS Code server) do you then have to set ”remote.SSH.useLocalServer": “false” once before it will start working again?

KasperSkytte commented 2 years ago

@KasperSkytte That’s what I suspected too. Curious if when you reboot the VM (thus stopping the remote VS Code server) do you then have to set ”remote.SSH.useLocalServer": “false” once before it will start working again?

@webvictim I deleted the VM right after the post. Now I started a new one (not entirely identical to the first one) and everything works fine no matter the setting, same for all other hosts I usually connect to. Odd. I tried killing the local code-server process, no difference.

KasperSkytte commented 2 years ago

Rebooting the workstation I connect from makes no difference either

webvictim commented 2 years ago

OK, I misunderstood that you were connecting from VM -> code server on workstation rather than workstation -> code server on VM. Either way, it's odd that it works fine with ”remote.SSH.useLocalServer": “true" after connecting once with ”remote.SSH.useLocalServer": “false”.

@tanhakabir Can you think of a reason why the first connection might be different? If we can figure out what's changing, we can probably fix Teleport to handle it and improve integration all round.

KasperSkytte commented 2 years ago

@webvictim Yes, maybe I was unclear, sorry. It would be awesome to get the full teleport feature set when working through VSCode remotely if things will work out nicely with tsh instead of ssh. I rarely use a terminal alone, it's just a pain doing any larger editing of files or doing development without a sophisticated IDE like VSCode.

tanhakabir commented 2 years ago

it's odd that it works fine with ”remote.SSH.useLocalServer": “true" after connecting once with ”remote.SSH.useLocalServer": “false”.

That's really odd. @KasperSkytte do you have logs of when it is successful with ”remote.SSH.useLocalServer": “true"?

KasperSkytte commented 2 years ago

@tanhakabir I can reproduce again now. It's not clear to me what the conditions are when it "just works" or when it doesn't. But here are the logs from when it correctly establishes connection with ”remote.SSH.useLocalServer": “true" after an initial connection with it set to false

[20:48:46.627] Log Level: 2
[20:48:46.628] remote-ssh@0.66.0
[20:48:46.628] linux x64
[20:48:46.629] SSH Resolver called for "ssh-remote+ubuntu20server.teleport.domain.com", attempt 1
[20:48:46.629] "remote.SSH.useLocalServer": true
[20:48:46.630] "remote.SSH.path": undefined
[20:48:46.630] "remote.SSH.configFile": undefined
[20:48:46.630] "remote.SSH.useFlock": true
[20:48:46.630] "remote.SSH.lockfilesInTmp": false
[20:48:46.630] "remote.SSH.localServerDownload": auto
[20:48:46.630] "remote.SSH.remoteServerListenOnSocket": false
[20:48:46.631] "remote.SSH.showLoginTerminal": false
[20:48:46.631] "remote.SSH.defaultExtensions": ["ms-vscode-remote.remote-containers","ms-vscode-remote.remote-ssh","ms-azuretools.vscode-docker","mhutchie.git-graph","eamodio.gitlens","tht13.html-preview-vscode","yzane.markdown-pdf","ms-vscode.live-server","christian-kohler.path-intellisense","timonwong.shellcheck","ms-python.vscode-pylance","ms-python.python","Ikuyadeu.r","RDebugger.r-debugger","TianyiShi.rmarkdown","mechatroner.rainbow-csv","janisdd.vscode-edit-csv","GrapeCity.gc-excelviewer","redhat.ansible","tomoki1207.pdf"]
[20:48:46.631] "remote.SSH.loglevel": 2
[20:48:46.631] SSH Resolver called for host: ubuntu20server.teleport.domain.com
[20:48:46.631] Setting up SSH remote "ubuntu20server.teleport.domain.com"
[20:48:46.635] Acquiring local install lock: /tmp/vscode-remote-ssh-5ba6fc6f-install.lock
[20:48:46.641] Looking for existing server data file at /home/user/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-5ba6fc6f-83bd43bc519d15e50c4272c6cf5c1479df196a4d-0.66.0/data.json
[20:48:46.642] Using commit id "83bd43bc519d15e50c4272c6cf5c1479df196a4d" and quality "stable" for server
[20:48:46.643] Install and start server if needed
[20:48:46.646] PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
[20:48:46.646] Checking ssh with "ssh -V"
[20:48:46.650] > OpenSSH_8.4p1 Ubuntu-5ubuntu1.2, OpenSSL 1.1.1j  16 Feb 2021

[20:48:46.657] askpass server listening on /run/user/1000/vscode-ssh-askpass-2ba9464e2390ed8c913c0883940ff84960b44ac5.sock
[20:48:46.658] Spawning local server with {"serverId":1,"ipcHandlePath":"/run/user/1000/vscode-ssh-askpass-4aa9103c487bd61c215b7310dbd1762d97e1daab.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","33639","-o","ConnectTimeout=15","ubuntu20server.teleport.domain.com","bash"],"dataFilePath":"/home/user/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-5ba6fc6f-83bd43bc519d15e50c4272c6cf5c1479df196a4d-0.66.0/data.json"}
[20:48:46.658] Local server env: {"DISPLAY":":1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.66.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/usr/share/code/code","VSCODE_SSH_ASKPASS_MAIN":"/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.66.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/run/user/1000/vscode-ssh-askpass-2ba9464e2390ed8c913c0883940ff84960b44ac5.sock"}
[20:48:46.659] Spawned 28821
[20:48:46.702] > local-server-1> Spawned ssh, pid=28829
[20:48:46.703] stderr> OpenSSH_8.4p1 Ubuntu-5ubuntu1.2, OpenSSL 1.1.1j  16 Feb 2021
[20:48:46.952] stderr> Authenticated to ubuntu20server.teleport.domain.com (via proxy).
[20:48:47.132] > ready: e3687e1569d1
[20:48:47.158] > Linux 5.4.0-105-generic #119-Ubuntu SMP Mon Mar 7 18:49:24 UTC 2022
[20:48:47.158] Platform: linux
[20:48:47.226] > e3687e1569d1: running
[20:48:47.251] > Acquiring lock on /home/user/.vscode-server/bin/83bd43bc519d15e50c4272c6cf5c1479df196a4d/vscode-remote-lock.kapper.83bd43bc519d15e50c4272c6cf5c1479df196a4d
[20:48:47.255] > Found existing installation at /home/user/.vscode-server/bin/83bd43bc519d15e50c4272c6cf5c1479df196a4d...
> Checking /home/user/.vscode-server/.83bd43bc519d15e50c4272c6cf5c1479df196a4d.log and /home/user/.vscode-server/.83bd43bc519d15e50c4272c6cf5c1479df196a4d.pid for a running server
[20:48:47.256] > Looking for server with pid: 2075
[20:48:47.269] > Found running server...
> Installing extensions...
[20:48:47.546] > Installing extensions...
[20:48:47.752] > Extension 'ms-vscode-remote.remote-containers' v0.224.3 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'ms-vscode-remote.remote-containers@1.2.3'.
> Extension 'ms-vscode-remote.remote-ssh' v0.76.1 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'ms-vscode-remote.remote-ssh@1.2.3'.
> Extension 'ms-azuretools.vscode-docker' v1.21.0 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'ms-azuretools.vscode-docker@1.2.3'.
> Extension 'mhutchie.git-graph' v1.30.0 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'mhutchie.git-graph@1.2.3'.
> Extension 'eamodio.gitlens' v12.0.5 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'eamodio.gitlens@1.2.3'.
> Extension 'tht13.html-preview-vscode' v0.2.5 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'tht13.html-preview-vscode@1.2.3'.
> Extension 'yzane.markdown-pdf' v1.4.4 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'yzane.markdown-pdf@1.2.3'.
> Extension 'ms-vscode.live-server' v0.2.12 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'ms-vscode.live-server@1.2.3'.
> Extension 'christian-kohler.path-intellisense' v2.8.0 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'christian-kohler.path-intellisense@1.2.3'.
> Extension 'timonwong.shellcheck' v0.18.9 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'timonwong.shellcheck@1.2.3'.
> Extension 'ms-python.vscode-pylance' v2022.3.3 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'ms-python.vscode-pylance@1.2.3'.
> Extension 'ms-python.python' v2022.2.1924087327 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'ms-python.python@1.2.3'.
> Extension 'ikuyadeu.r' v2.4.0 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'ikuyadeu.r@1.2.3'.
> Extension 'rdebugger.r-debugger' v0.4.7 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'rdebugger.r-debugger@1.2.3'.
> Extension 'tianyishi.rmarkdown' v0.0.14 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'tianyishi.rmarkdown@1.2.3'.
> Extension 'mechatroner.rainbow-csv' v2.1.0 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'mechatroner.rainbow-csv@1.2.3'.
> Extension 'janisdd.vscode-edit-csv' v0.6.6 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'janisdd.vscode-edit-csv@1.2.3'.
> Extension 'grapecity.gc-excelviewer' v4.2.53 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'grapecity.gc-excelviewer@1.2.3'.
> Extension 'redhat.ansible' v0.8.1 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'redhat.ansible@1.2.3'.
> Extension 'tomoki1207.pdf' v1.2.0 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'tomoki1207.pdf@1.2.3'.
>  
> *
> * Reminder: You may only use this software with Visual Studio family products,
> * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
> *
>  
> Checking server status on port 34005 with wget
> e3687e1569d1: start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn==34005==
> osReleaseId==ubuntu==
> arch==x86_64==
> tmpDir==/tmp==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> extInstallTime==393==
> serverStartTime====
> connectionToken==a11a11a1-aa11-1aa1-aa11-1111a111111a==
> e3687e1569d1: end
[20:48:47.752] Received install output: 
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==34005==
osReleaseId==ubuntu==
arch==x86_64==
tmpDir==/tmp==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
extInstallTime==393==
serverStartTime====
connectionToken==a11a11a1-aa11-1aa1-aa11-1111a111111a==

[20:48:47.752] Remote server is listening on 34005
[20:48:47.753] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":34005},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/tmp","platform":"linux","connectionToken":"a11a11a1-aa11-1aa1-aa11-1111a111111a"},"extInstallTime":393,"installUnpackCode":""}
[20:48:47.753] Persisting server connection details to /home/user/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-5ba6fc6f-83bd43bc519d15e50c4272c6cf5c1479df196a4d-0.66.0/data.json
[20:48:47.755] Starting forwarding server. localPort 40775 -> socksPort 33639 -> remotePort 34005
[20:48:47.755] Forwarding server listening on 40775
[20:48:47.755] Waiting for ssh tunnel to be ready
[20:48:47.756] [Forwarding server 40775] Got connection 0
[20:48:47.757] Tunneled 34005 to local port 40775
[20:48:47.757] Resolved "ssh-remote+ubuntu20server.teleport.domain.com" to "127.0.0.1:40775"
[20:48:47.765] ------

[20:48:47.817] [Forwarding server 40775] Got connection 1
[20:48:47.913] [Forwarding server 40775] Got connection 2
tanhakabir commented 2 years ago

It's really odd to me that on the second run it doesn't seem like the script is being echo'd as seen in your original logs. @webvictim I really can't tell much more here since I don't know how tsh works but I would expect that how scripts work when they're getting piped into a process connected to a remote to be consistently the same. We run the exact same flow every time you go through and try to connect.

webvictim commented 2 years ago

I'm curious whether the script runs curl or wget or something to download the server on the remote machine if it's not present, or whether it transfers it down the SSH connection from the local machine.

tanhakabir commented 2 years ago

@webvictim it does all of the above. It checks if you have curl or wget on your remote machine and tries to download the vscode server with that. If it's not available or if cur or wget error out too many times (I think >3) then it downloads the vscode server on your local machine and transfers the file over.

webvictim commented 2 years ago

Interesting. @KasperSkytte did you have curl or wget on your server? I'm wondering whether maybe the file transfer part could be causing the issue.

tanhakabir commented 2 years ago

It's unlikely to be the issue because in the first logs he shared it shows how the script he runs is just echo'd. If it gets as far as the wget check then he would have no issues between useLocalServer: false and true.

The main issue I see here is from the first logs where the script is being echo'd. That's not common behavior with ssh.

KasperSkytte commented 2 years ago

@webvictim Yes both curl and wget are installed.

KasperSkytte commented 2 years ago

So, what do we do here? :)

tanhakabir commented 2 years ago

@KasperSkytte at this time I suppose still keep using the workaround. I can investigate the exact differences between tsh and ssh in a later time but currently I don't know why this is an issue

webvictim commented 2 years ago

From the Teleport side we just don't have the time to be able to reproduce this and work on fixing it, I'm sorry. I'd like to but it's just not a priority when there's already a documented workaround.

cadwallion commented 2 years ago

Greetings! I've been investigating this issue for my own adoption of Teleport into developer environments, and I have been running into the same problem described here. What I discovered is that the remote.SSH.useLocalServer setting is only indirectly the workaround. What matters (and the reason that remote.SSH.useLocalServer: false helps) is that remote.SSH.remotePlatform gets written with a host-specific entry for the operating system of the remote host. When you set "remote.SSH.useLocalServer": false and attempt to connect the first time, you are prompted with what operating system the remote host is running. Once you select that, an entry is added in remote.SSH.remotePlatform for that FQDN like so:

{
  "remote.SSH.remotePlatform": {
    "remote-host-a1b2c3d4.example.dev.teleport.core.example.com": "linux"
  }
}

As long as that entry in remote.SSH.remotePlatform stays in the Settings JSON, useLocalServer value doesn't seem to matter. Having a running vscode agent on the remote host doesn't let us remove this entry either, as the following scenario is reproducible:

This is somewhat perplexing, because the local server option is able to detect on my remote hosts the OS (Linux), shell (bash), and bash version (4.4.20(1)-release), so having an entry for the OS in the settings be the solution seems odd.

Unfortunately for me, I cannot use the workaround because my entire system is based upon ephemeral compute with programmatically derived hostnames, which means I'd not only have to advise my users that they set remote.SSH.useLocalServer to false and have to manually select the OS every time they spin up a new environment, but the remotePlatform entries would balloon over time in the settings JSON. On a whim, I tried altering the entry in remote.SSH.remotePlatform to *.teleport.core.example.com (domain name of Teleport cluster swapped for security purposes), but unfortunately no luck there as I'm guessing it's looking for an exact match.

I'm going to continue poking at this, but I wanted to document what I've found so far for future investigations and to see if anyone has any new ideas based upon this.

vscodenpa commented 1 year ago

This issue is being closed to keep the number of issues in our inbox on a manageable level, we are closing issues that are not going to be addressed in the foreseeable future: We look at the number of votes the issue has received and the number of duplicate issues filed. More details here. If you disagree and feel that this issue is crucial: We are happy to listen and to reconsider.

If you wonder what we are up to, please see our roadmap and issue reporting guidelines.

Thanks for your understanding and happy coding!