search

microsoft / vscode-remote-release

Visual Studio Code Remote Development: Open any folder in WSL, in a Docker container, or on a remote machine using SSH and take advantage of VS Code's full feature set.
https://aka.ms/vscode-remote
Other
3.63k stars 282 forks source link

[Remote-SSH Bug]: Remote SSH with RemoteCommand WSL to Windows Workstation host broken on v0.114.x #10305

Open joeterlecki opened 6 days ago

joeterlecki commented 6 days ago

I use remote SSH extensively to connect to my workstation via SSH. I have 2 hosts in my config file:

  1. windows 11
  2. windows 11 wsl

Prior to updating the SSH extension from 0.113.x to v0.114.x RemoteCommand used to work to get a WSL tty. Then SSH connection obviously starts with powershell but once a connection is made the RemoteCommand of wsl would grant me a wsl tty and I could use my WSL instance for development remotely.

Here is my config file for reference: `Host werkstation-pwsh HostName 100.109.147.22 User joept RemoteCommand powershell.exe RequestTTY yes ServerAliveInterval 60 ServerAliveCountMax 10

Host werkstation-wsl HostName 100.109.147.22 User joept RemoteCommand wsl RequestTTY yes ServerAliveInterval 60 ServerAliveCountMax 10`

However, after trying all 0.14 versions I can no longer get a TTY to wsl through ssh tunnel on the extension. As you can see in the log below, the RemoteCommand is not picked up but 'sh' command cant be found as it tries to run shell while still exec'ed into powershell.

I have used this SSH method for years over terminal and discovered its capability also existed with the SSH extension ~2 years ago so it has been a functioning native feature for some time.

I have tried the following before reverting back to last working version:

  1. reinstalled vscode
  2. deleted all remote .vscode-server configurations
  3. reinstalled all extensions.
  4. tried every 0.14.x version
  5. fresh WSL instance. (Ubuntu)

 [21:52:31.100] "install" terminal command done [21:52:31.101] Install terminal quit with output:
[21:52:31.101] Received install output:
[21:52:31.102] Exec server for ssh-remote+werkstation-wsl failed: Error [21:52:31.102] Existing exec server for ssh-remote+werkstation-wsl errored (Error) [21:52:31.104] Initizing new exec server for ssh-remote+werkstation-wsl [21:52:31.104] Using commit id "38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40" and quality "stable" for server [21:52:31.106] Error opening exec server for ssh-remote+werkstation-wsl: Error [21:52:31.114] Script variables: { "InstallExitCode.24": "AlreadyInProgress", "InstallExitCode.25": "ServerDownloadFailed", "InstallExitCode.26": "NoDownloaderAvailable", "InstallExitCode.27": "UnsupportedArch", "InstallExitCode.28": "StatusCheckFailed", "InstallExitCode.29": "NeedInsidersArch", "InstallExitCode.30": "NoDownloaderAvailableForStatusCheck", "InstallExitCode.31": "ServerTransferFailed", "InstallExitCode.32": "ServerFailedToStart", "InstallExitCode.33": "NeedInsidersWindows", "InstallExitCode.34": "CreateInstallDirFailed", "InstallExitCode.35": "UnsupportedPlatform", "InstallExitCode.36": "ServerTerminatedCVE20201416", "InstallExitCode.37": "UnpackFailed", "InstallExitCode.38": "ChangeDirFailed", "InstallExitCode.AlreadyInProgress": "24", "InstallExitCode.ServerDownloadFailed": "25", "InstallExitCode.NoDownloaderAvailable": "26", "InstallExitCode.NoDownloaderAvailableForStatusCheck": "30", "InstallExitCode.UnsupportedArch": "27", "InstallExitCode.StatusCheckFailed": "28", "InstallExitCode.NeedInsidersArch": "29", "InstallExitCode.ServerTransferFailed": "31", "InstallExitCode.ServerFailedToStart": "32", "InstallExitCode.NeedInsidersWindows": "33", "InstallExitCode.CreateInstallDirFailed": "34", "InstallExitCode.UnsupportedPlatform": "35", "InstallExitCode.ServerTerminatedCVE20201416": "36", "InstallExitCode.UnpackFailed": "37", "InstallExitCode.ChangeDirFailed": "38", "InstallUnpackCode.Success": "success", "InstallUnpackCode.Error": "error", "InstallUnpackCode.MissingFiles": "missingFiles", "uuid": "7f9e90973dcb", "startMarker": "7f9e90973dcb: running", "commitId": "38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40", "quality": "stable", "token": "aa1aaaaa-1a11-111a-1a11-a11a1a11aaa1", "vscodeAgentFolder": "$HOME/.vscode-server", "allowClientDownload": "1", "forceClientDownload": "0", "cliNameInArchive": "code", "ignoreWgetConfigFlag": " --no-config ", "ignoreCurlConfigFlag": " --disable ", "wgetTriesSegment": "--tries=1", "listenArgs": "--on-host=127.0.0.1 --on-port", "getDownloadServerStartTrigger": "7f9e90973dcb:trigger_server_download", "getDownloadServerEndTrigger": "7f9e90973dcb:trigger_server_download_end", "getProgressDownloading": "7f9e90973dcb%%1%%", "getProgressInstalling": "7f9e90973dcb%%2%%" } [21:52:31.118] Install and start server if needed [21:52:31.131] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 51274 -o RemoteCommand=none "werkstation-wsl" sh [21:52:31.133] Generated SSH command: 'type "C:\Users\joept\AppData\Local\Temp\vscode-linux-multi-line-command-werkstation-wsl-339229003.sh" | "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 51274 -o RemoteCommand=none "werkstation-wsl" sh' [21:52:31.134] Terminal shell path: C:\Windows\System32\cmd.exe [21:52:31.501] > [?9001h[?1004h [21:52:31.501] Got some output, clearing connection timeout [21:52:31.891] > sh : The term 'sh' is not recognized as the name of a cmdlet, function, script file, or operable

]0;C:\Windows\System32\cmd.exeprogram. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1

  • sh
  • ~~
    • CategoryInfo : ObjectNotFound: (sh:String) [], CommandNotFoundException
    • FullyQualifiedErrorId : CommandNotFoundException

[21:52:33.178] "install" terminal command done [21:52:33.178] Install terminal quit with output:
[21:52:33.179] Received install output:
[21:52:33.179] Exec server for ssh-remote+werkstation-wsl failed: Error [21:52:33.179] Error opening exec server for ssh-remote+werkstation-wsl: Error

joshspicer commented 5 days ago

Thanks for the details. I'm not sure the problem yet but given your report (and the changes that went in to the 0.114 release) this definitely sounds like a bug.

joshspicer commented 5 days ago

@joeterlecki Would you be able to re-attach your entire log file? I've noticed that you only shared a subset of the log (i'd like to see your settings toward the top of the log, for example).

joshspicer commented 5 days ago

At first glance I see that on my local machine I do have bash on my path but not sh, which likely explains the behavior change

Image

joeterlecki commented 5 days ago

Yeah sure thing, is there a better way to get log file for this output? I just copied the STDOUT from the terminal/output when it attempts to connect.

In response to your "first glance" there is no "sh" on my path either. My suspicion with with 0.14 changes, is the SHELL for starting the openssh session changed or possibly trying a linux shell execution, but not sure how to validate that assumption.

I would like to get you a log of both the latest 0.13 and .0.14 extension versions for comparison. Is there also a way to output more verbose and debug logs for you?

I will try and test the reverse and try SSH Extension on workstation -> surface laptop and see if the behavior is the same to hopefully rule out ARM host issue

joshspicer commented 5 days ago

is there a better way to get log file for this output? I just copied the STDOUT from the terminal/output when it attempts to connect.

That should do it (the command in the command palette is Remote-SSH: Show Log if you need it) There's also a setting for changing the verbosity of the logs. I don't think that will be necessary for me but also won't hurt!

joeterlecki commented 5 days ago

Here is the complete logs for both the working 0.113 and 0.114 logs

Let me know if there is more info, i would be glad to get for you.

Thank you again.

remote-ssh-0.13.1-log.txt remote-ssh-0.114.3-log.txt

joeterlecki commented 5 days ago

Looking at both logs, this appears to be the difference when spawning th SSH Command

on 113 [22:28:10.839] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 65475 -o RemoteCommand=none "werkstation-wsl" bash

and on 114 [22:32:07.386] Generated SSH command: 'type "C:\Users\joept\AppData\Local\Temp\vscode-linux-multi-line-command-werkstation-wsl-975832064.sh" | "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 65529 -o RemoteCommand=none "werkstation-wsl" sh'

Looks like its trying to run sh instead of bash which crashes the remote connection attempt

WHen i double check my windows powershell ssh I see this line whiich shows that powershell as the command for the TTY [22:40:58.251] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 49244 -o RemoteCommand=none "werkstation-pwsh" powershell

When I set up the SSH hosts, I picked "Linux" for the werkstation-wsl host, so I'm taking a guess that from 113 to 114 changed the spawn shell from BASH to SH TTY which I would expect to crash as windows doesnt have SH as a shell, but "bash" spawns the wsl shell TTY

joshspicer commented 4 days ago

Thanks for the logs. I was indeed curious what platform you were selecting when connecting to werkstation-wsl. Have you tried selecting windows as that platform? While I understand the confusion here, from the extension's perspective you're still connecting to a Windows machine (then opening a wsl shell).

joeterlecki commented 4 days ago

Yeah results in the same behavior.

I thought about that too during my initial investigation.

here is a snippet of the config right now

Screenshot 2024-09-25 142722

As you can see its set for linux.

I also set the same host as a direct ip instead of hostname to ensure it still fails the same way.

What was the significant change from 113 to 114 that change the shell from bash to sh?

Is this a change that could potentially be reverted? It had been pretty awesome to just remote into my WSL instance on my workstation and go to town

joeterlecki commented 15 hours ago

I have done some more digging and I think its safe to assume this is maybe not a bug with ssh itself.

After setting at remote ssh profile host as linux or windows, it determines what is ran at successful connection. If the host is linux it runs the openssh command as follows with "sh" "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 65529 -o RemoteCommand=none "werkstation-wsl" sh'

Obviously this is failing because sh is not a windows cmd and previous versions defaulted to "bash"

If you think that the change from 113 to 114 requires this, and it is not a bug because of the special situation where I'm using it to connect to a wsl instance, I understand, and you can close this out or revert to bash, but I don't have the context of the changes to 114.

I did find a workaround for 114:

  1. create an profile.ps1 at the sytsem level - C:\Windows\System32\WindowsPowerShell\v1.0\profile.ps1
  2. create an alias for bash as sh - Set-Alias sh bash
  3. Works 10\10

Below is the log output with the ALIAS. Thanks again for your support and help. It would be cool to not need an alias but if the "sh" is needed to support systems without bash such as alpine, its all good. Or maybe have a setting to pick which shell to run with the ssh command (zsh, FSH, bash) ect...

`[14:33:40.343] Log Level: 2 [14:33:40.353] SSH Resolver called for "ssh-remote+werkstation-wsl", attempt 1 [14:33:40.354] "remote.SSH.useLocalServer": false [14:33:40.354] "remote.SSH.useExecServer": true [14:33:40.354] "remote.SSH.showLoginTerminal": false [14:33:40.354] "remote.SSH.remotePlatform": {"192.168.1.48":"linux","werkstation-wsl":"linux","werkstation-pwsh":"windows","100.109.147.22":"linux"} [14:33:40.354] "remote.SSH.path": undefined [14:33:40.354] "remote.SSH.configFile": undefined [14:33:40.354] "remote.SSH.useFlock": true [14:33:40.354] "remote.SSH.lockfilesInTmp": false [14:33:40.355] "remote.SSH.localServerDownload": auto [14:33:40.355] "remote.SSH.remoteServerListenOnSocket": false [14:33:40.355] "remote.SSH.showLoginTerminal": false [14:33:40.355] "remote.SSH.defaultExtensions": [] [14:33:40.355] "remote.SSH.loglevel": 2 [14:33:40.355] "remote.SSH.enableDynamicForwarding": true [14:33:40.355] "remote.SSH.enableRemoteCommand": false [14:33:40.355] "remote.SSH.serverPickPortsFromRange": {} [14:33:40.355] "remote.SSH.serverInstallPath": {} [14:33:40.355] "remote.SSH.permitPtyAllocation": false [14:33:40.355] "remote.SSH.preferredLocalPortRange: undefined [14:33:40.355] "remote.SSH.useCurlAndWgetConfigurationFiles: false [14:33:40.363] VS Code version: 1.93.1 [14:33:40.363] Remote-SSH version: remote-ssh@0.114.3 [14:33:40.363] win32 arm64 [14:33:40.365] SSH Resolver called for host: werkstation-wsl [14:33:40.365] Setting up SSH remote "werkstation-wsl" [14:33:40.368] Using commit id "38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40" and quality "stable" for server [14:33:40.373] Script variables: { "InstallExitCode.24": "AlreadyInProgress", "InstallExitCode.25": "ServerDownloadFailed", "InstallExitCode.26": "NoDownloaderAvailable", "InstallExitCode.27": "UnsupportedArch", "InstallExitCode.28": "StatusCheckFailed", "InstallExitCode.29": "NeedInsidersArch", "InstallExitCode.30": "NoDownloaderAvailableForStatusCheck", "InstallExitCode.31": "ServerTransferFailed", "InstallExitCode.32": "ServerFailedToStart", "InstallExitCode.33": "NeedInsidersWindows", "InstallExitCode.34": "CreateInstallDirFailed", "InstallExitCode.35": "UnsupportedPlatform", "InstallExitCode.36": "ServerTerminatedCVE20201416", "InstallExitCode.37": "UnpackFailed", "InstallExitCode.38": "ChangeDirFailed", "InstallExitCode.AlreadyInProgress": "24", "InstallExitCode.ServerDownloadFailed": "25", "InstallExitCode.NoDownloaderAvailable": "26", "InstallExitCode.NoDownloaderAvailableForStatusCheck": "30", "InstallExitCode.UnsupportedArch": "27", "InstallExitCode.StatusCheckFailed": "28", "InstallExitCode.NeedInsidersArch": "29", "InstallExitCode.ServerTransferFailed": "31", "InstallExitCode.ServerFailedToStart": "32", "InstallExitCode.NeedInsidersWindows": "33", "InstallExitCode.CreateInstallDirFailed": "34", "InstallExitCode.UnsupportedPlatform": "35", "InstallExitCode.ServerTerminatedCVE20201416": "36", "InstallExitCode.UnpackFailed": "37", "InstallExitCode.ChangeDirFailed": "38", "InstallUnpackCode.Success": "success", "InstallUnpackCode.Error": "error", "InstallUnpackCode.MissingFiles": "missingFiles", "uuid": "7f56fba23bec", "startMarker": "7f56fba23bec: running", "commitId": "38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40", "quality": "stable", "token": "1111a111-a111-1a11-1a1a-1111aa1aa1a1", "vscodeAgentFolder": "$HOME/.vscode-server", "allowClientDownload": "1", "forceClientDownload": "0", "cliNameInArchive": "code", "ignoreWgetConfigFlag": " --no-config ", "ignoreCurlConfigFlag": " --disable ", "wgetTriesSegment": "--tries=1", "listenArgs": "--on-host=127.0.0.1 --on-port", "getDownloadServerStartTrigger": "7f56fba23bec:trigger_server_download", "getDownloadServerEndTrigger": "7f56fba23bec:trigger_server_download_end", "getProgressDownloading": "7f56fba23bec%%1%%", "getProgressInstalling": "7f56fba23bec%%2%%" } [14:33:40.375] Install and start server if needed [14:33:40.380] Checking ssh with "C:\Windows\system32\ssh.exe -V" [14:33:40.381] Got error from ssh: spawn C:\Windows\system32\ssh.exe ENOENT [14:33:40.381] Checking ssh with "C:\Windows\ssh.exe -V" [14:33:40.382] Got error from ssh: spawn C:\Windows\ssh.exe ENOENT [14:33:40.382] Checking ssh with "C:\Windows\System32\Wbem\ssh.exe -V" [14:33:40.383] Got error from ssh: spawn C:\Windows\System32\Wbem\ssh.exe ENOENT [14:33:40.383] Checking ssh with "C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe -V" [14:33:40.384] Got error from ssh: spawn C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe ENOENT [14:33:40.384] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V" [14:33:40.411] > OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2

[14:33:40.414] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 63527 -o RemoteCommand=none "werkstation-wsl" sh [14:33:40.415] Generated SSH command: 'type "C:\Users\joept\AppData\Local\Temp\vscode-linux-multi-line-command-werkstation-wsl-282354640.sh" | "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 63527 -o RemoteCommand=none "werkstation-wsl" sh' [14:33:40.416] Terminal shell path: C:\Windows\System32\cmd.exe [14:33:40.695] > [?9001h[?1004h [14:33:40.695] Got some output, clearing connection timeout [14:33:41.232] > 7f56fba23bec: running

]0;C:\Windows\System32\cmd.exeScript executing under PID: 26579 Found existing installation at /home/jterlecki/.vscode-server... Starting VS Code CLI... printenv: SHELL=/usr/bin/zsh WSL2_GUI_APPS_ENABLED=1 WSL_DISTRO_NAME=Ubuntu-24.04 NAME=werkstation PWD=/mnt/c/Users/joept LOGNAME=jterlecki HOME=/home/jterlecki LANG=C.UTF-8 WSL_INTEROP=/run/WSL/26573_interop WAYLAND_DISPLAY=wayland-0 TERM=xterm-256color USER=jterlecki DISPLAY=:0 SHLVL=1 XDG_RUNTIME_DIR=/run/user/1000/ WSLENV= XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/Tailscale/:/mnt/c/Program Files (x86)/Microsoft SQL Server/160/DTS/Binn/:/mnt/c/Program Files (x86)/Microsoft SQL Server/160/Tools/Binn/:/mnt/c/Program Files/Microsoft SQL Server/160/Tools/Binn/:/mnt/c/Program Fi iles/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn/:/mnt/c/Program Files/Mi icrosoft SQL Server/160/DTS/Binn/:/mnt/c/WINDOWS/system32/config/systemprofile/Ap ppData/Local/Microsoft/WindowsApps:/mnt/c/Users/joept/AppData/Local/Microsoft/Win ndowsApps:/mnt/c/Users/joept/.dotnet/tools:/mnt/c/Users/joept/AppData/Local/Progr rams/Microsoft VS Code/bin:/mnt/c/Users/joept/AppData/Local/JetBrains/Toolbox/scr ripts:/mnt/c/Users/joept/AppData/Local/Microsoft/WinGet/Packages/Git.MinGit_Micro osoft.Winget.Source_8wekyb3d8bbwe/cmd:/snap/bin:/home/jterlecki/.local/share/JetB Brains/Toolbox/scripts DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus HOSTTYPE=x86_64 PULSESERVER=unix:/mnt/wslg/PulseServer =/usr/bin/printenv [14:33:41.239] > Removing old logfile at /home/jterlecki/.vscode-server/.cli.38c31bc77e0dd6ae88a4 4e9cc93428cc27a56ba40.log Spawned remote CLI: 26602 Waiting for server log... [14:33:41.263] > 7f56fba23bec: start listeningOn==127.0.0.1:36533== osReleaseId==ubuntu== arch==x86_64== vscodeArch==x64== bitness==64== tmpDir==/run/user/1000/== platform==linux== unpackResult==== didLocalDownload==0== downloadTime==== installTime==== serverStartTime==35== execServerToken==1111a111-a111-1a11-1a1a-1111aa1aa1a1== platformDownloadPath==cli-alpine-x64== 7f56fba23bec: end [14:33:41.263] Received install output: listeningOn==127.0.0.1:36533== osReleaseId==ubuntu== arch==x86_64== vscodeArch==x64== bitness==64== tmpDir==/run/user/1000/== platform==linux== unpackResult==== didLocalDownload==0== downloadTime==== installTime==== serverStartTime==35== execServerToken==1111a111-a111-1a11-1a1a-1111aa1aa1a1== platformDownloadPath==cli-alpine-x64==

[14:33:41.265] Remote server is listening on port 36533 [14:33:41.265] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":36533},"osReleaseId":"ubuntu","arch":"x86_64","tmpDir":"/run/user/1000/","platform":"linux","execServerToken":"1111a111-a111-1a11-1a1a-1111aa1aa1a1"},"serverStartTime":35,"installUnpackCode":""} [14:33:41.283] Starting forwarding server. local port 63529 -> socksPort 63527 -> remotePort 36533 [14:33:41.286] Forwarding server listening on port 63529 [14:33:41.286] Waiting for ssh tunnel to be ready [14:33:41.290] > [14:33:41.291] [Forwarding server port 63529] Got connection 0 [14:33:41.296] Tunneled port 36533 to local port 63529 [14:33:41.297] Resolved "ssh-remote+werkstation-wsl" to "port 63529" [14:33:41.318] Initizing new exec server for ssh-remote+werkstation-wsl [14:33:41.318] Resolving exec server at port 63529 [14:33:41.347] [Forwarding server port 63529] Got connection 1 [14:33:42.408] Exec server for ssh-remote+werkstation-wsl created and cached [14:33:42.434] ------

[14:33:42.493] [server] Checking /home/jterlecki/.vscode-server/cli/servers/Stable-38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40/log.txt and /home/jterlecki/.vscode-server/cli/servers/Stable-38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40/pid.txt for a running server... [14:33:42.494] [server] Found running server (pid=25143) [14:33:43.639] Opening exec server for ssh-remote+werkstation-wsl [14:33:43.729] Opening exec server for ssh-remote+werkstation-wsl [14:33:43.748] Verified and reusing cached exec server for ssh-remote+werkstation-wsl [14:33:43.760] Verified and reusing cached exec server for ssh-remote+werkstation-wsl [14:33:43.769] Opening exec server for ssh-remote+werkstation-wsl [14:33:43.774] Verified and reusing cached exec server for ssh-remote+werkstation-wsl`