search

microsoft / vscode-remote-release

Visual Studio Code Remote Development: Open any folder in WSL, in a Docker container, or on a remote machine using SSH and take advantage of VS Code's full feature set.
https://aka.ms/vscode-remote
Other
3.68k stars 296 forks source link

[Remote-SSH Bug]: Hangs after "Copy server to host" #10345

Closed arghness closed 1 month ago

arghness commented 1 month ago

Is there an existing issue for this bug?

Required Troubleshooting Steps

Connect Locally

It connects successfully

->

No response

Expected Behavior

I expect that after the server is copied, I should be able to use the remote host.

Steps To Reproduce

  1. Tested on Rocky Linux 9, Rocky Linux 8, CentOS 7.
  2. SSH agent configured and working, SSH config: 
    Host rocky9-product-master
HostName host122
Port 25060
User myuser
ForwardAgent yes
  3. Delete the vscode server install directory on the remote host for a clean start.
  4. Connect to host.
  5. Connects, appears to copy the server to the host successfully, then hangs and requires a disconnect. Reconnecting repeats the process and hangs again.

Remote-SSH Log

Remote-SSH Log

``` [14:09:02.742] Log Level: 2 [14:09:02.748] VS Code version: 1.94.0 [14:09:02.748] Remote-SSH version: remote-ssh@0.114.3 [14:09:02.748] win32 x64 [14:09:02.766] SSH Resolver called for "ssh-remote+rocky9-product-master", attempt 1 [14:09:02.766] "remote.SSH.useLocalServer": false [14:09:02.766] "remote.SSH.useExecServer": true [14:09:02.766] "remote.SSH.showLoginTerminal": false [14:09:02.767] "remote.SSH.remotePlatform": {"*-product-*":"linux","centos7-product-master":"linux","cuda-dev":"linux","rocky8-product-11.1":"linux","rocky8-product-master":"linux","rocky9-product-master":"linux","vmxscratch":"linux","obs":"linux","rocky8-product-12":"linux","rocky9-product-12":"linux","rocky8-product-11.0":"linux"} [14:09:02.767] "remote.SSH.path": undefined [14:09:02.767] "remote.SSH.configFile": undefined [14:09:02.767] "remote.SSH.useFlock": true [14:09:02.767] "remote.SSH.lockfilesInTmp": false [14:09:02.767] "remote.SSH.localServerDownload": auto [14:09:02.767] "remote.SSH.remoteServerListenOnSocket": false [14:09:02.767] "remote.SSH.showLoginTerminal": false [14:09:02.767] "remote.SSH.defaultExtensions": [] [14:09:02.767] "remote.SSH.loglevel": 2 [14:09:02.767] "remote.SSH.enableDynamicForwarding": true [14:09:02.767] "remote.SSH.enableRemoteCommand": false [14:09:02.767] "remote.SSH.serverPickPortsFromRange": {} [14:09:02.768] "remote.SSH.serverInstallPath": {"rocky9-product-master":"vscode_server/rocky9-dev-prod-master-myuser","rocky8-product-master":"vscode_server/rocky8-dev-prod-master-myuser","centos7-product-master":"vscode_server/centos7-dev-prod-master-myuser","rocky9-product-12":"vscode_server/rocky9-dev-prod-12-myuser","rocky8-product-12":"vscode_server/rocky8-dev-prod-12-myuser","centos7-product-12":"vscode_server/centos7-dev-prod-12-myuser","rocky8-product-11.1":"vscode_server/rocky8-dev-prod-11-1-myuser","centos7-product-11.1":"vscode_server/centos7-dev-prod-11-1-myuser","rocky8-product-11.0":"vscode_server/rocky8-dev-prod-11-0-myuser"} [14:09:02.768] "remote.SSH.permitPtyAllocation": false [14:09:02.771] "remote.SSH.preferredLocalPortRange: undefined [14:09:02.771] "remote.SSH.useCurlAndWgetConfigurationFiles: false [14:09:02.781] SSH Resolver called for host: rocky9-product-master [14:09:02.781] Setting up SSH remote "rocky9-product-master" [14:09:02.784] Using commit id "d78a74bcdfad14d5d3b1b782f87255d802b57511" and quality "stable" for server [14:09:02.786] Script variables: { "InstallExitCode.24": "AlreadyInProgress", "InstallExitCode.25": "ServerDownloadFailed", "InstallExitCode.26": "NoDownloaderAvailable", "InstallExitCode.27": "UnsupportedArch", "InstallExitCode.28": "StatusCheckFailed", "InstallExitCode.29": "NeedInsidersArch", "InstallExitCode.30": "NoDownloaderAvailableForStatusCheck", "InstallExitCode.31": "ServerTransferFailed", "InstallExitCode.32": "ServerFailedToStart", "InstallExitCode.33": "NeedInsidersWindows", "InstallExitCode.34": "CreateInstallDirFailed", "InstallExitCode.35": "UnsupportedPlatform", "InstallExitCode.36": "ServerTerminatedCVE20201416", "InstallExitCode.37": "UnpackFailed", "InstallExitCode.38": "ChangeDirFailed", "InstallExitCode.AlreadyInProgress": "24", "InstallExitCode.ServerDownloadFailed": "25", "InstallExitCode.NoDownloaderAvailable": "26", "InstallExitCode.NoDownloaderAvailableForStatusCheck": "30", "InstallExitCode.UnsupportedArch": "27", "InstallExitCode.StatusCheckFailed": "28", "InstallExitCode.NeedInsidersArch": "29", "InstallExitCode.ServerTransferFailed": "31", "InstallExitCode.ServerFailedToStart": "32", "InstallExitCode.NeedInsidersWindows": "33", "InstallExitCode.CreateInstallDirFailed": "34", "InstallExitCode.UnsupportedPlatform": "35", "InstallExitCode.ServerTerminatedCVE20201416": "36", "InstallExitCode.UnpackFailed": "37", "InstallExitCode.ChangeDirFailed": "38", "InstallUnpackCode.Success": "success", "InstallUnpackCode.Error": "error", "InstallUnpackCode.MissingFiles": "missingFiles", "uuid": "071dd51dd448", "startMarker": "071dd51dd448: running", "commitId": "d78a74bcdfad14d5d3b1b782f87255d802b57511", "quality": "stable", "token": "111a111a-1aa1-11a1-1aaa-1a111a111a11", "vscodeAgentFolder": "vscode_server/rocky9-dev-prod-master-myuser/.vscode-server", "allowClientDownload": "1", "forceClientDownload": "0", "cliNameInArchive": "code", "ignoreWgetConfigFlag": " --no-config ", "ignoreCurlConfigFlag": " --disable ", "wgetTriesSegment": "--tries=1", "listenArgs": "--on-host=127.0.0.1 --on-port", "getDownloadServerStartTrigger": "071dd51dd448:trigger_server_download", "getDownloadServerEndTrigger": "071dd51dd448:trigger_server_download_end", "getProgressDownloading": "071dd51dd448%%1%%", "getProgressInstalling": "071dd51dd448%%2%%" } [14:09:02.788] Install and start server if needed [14:09:02.790] Checking ssh with "C:\Program Files\Zulu\zulu-11\bin\ssh.exe -V" [14:09:02.792] Got error from ssh: spawn C:\Program Files\Zulu\zulu-11\bin\ssh.exe ENOENT [14:09:02.793] Checking ssh with "C:\Program Files\Zulu\zulu-8\bin\ssh.exe -V" [14:09:02.793] Got error from ssh: spawn C:\Program Files\Zulu\zulu-8\bin\ssh.exe ENOENT [14:09:02.794] Checking ssh with "C:\WINDOWS\system32\ssh.exe -V" [14:09:02.794] Got error from ssh: spawn C:\WINDOWS\system32\ssh.exe ENOENT [14:09:02.794] Checking ssh with "C:\WINDOWS\ssh.exe -V" [14:09:02.795] Got error from ssh: spawn C:\WINDOWS\ssh.exe ENOENT [14:09:02.795] Checking ssh with "C:\WINDOWS\System32\Wbem\ssh.exe -V" [14:09:02.796] Got error from ssh: spawn C:\WINDOWS\System32\Wbem\ssh.exe ENOENT [14:09:02.796] Checking ssh with "C:\WINDOWS\System32\WindowsPowerShell\v1.0\ssh.exe -V" [14:09:02.797] Got error from ssh: spawn C:\WINDOWS\System32\WindowsPowerShell\v1.0\ssh.exe ENOENT [14:09:02.797] Checking ssh with "C:\WINDOWS\System32\OpenSSH\ssh.exe -V" [14:09:02.827] > OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2 [14:09:02.831] Running script with connection command: "C:\WINDOWS\System32\OpenSSH\ssh.exe" -T -D 61375 "rocky9-product-master" sh [14:09:02.833] Generated SSH command: 'type "C:\Users\MYUSER~1\AppData\Local\Temp\vscode-linux-multi-line-command-rocky9-product-master-319638561.sh" | "C:\WINDOWS\System32\OpenSSH\ssh.exe" -T -D 61375 "rocky9-product-master" sh' [14:09:02.834] Terminal shell path: C:\WINDOWS\System32\cmd.exe [14:09:03.073] > [?9001h[?1004h [14:09:03.074] Got some output, clearing connection timeout [14:09:03.803] > 071dd51dd448: running > ]0;C:\WINDOWS\System32\cmd.exe [14:09:03.824] > Script executing under PID: 85259 > Installing to vscode_server/rocky9-dev-prod-master-myuser/.vscode-server... > 071dd51dd448%%1%% > Downloading with wget > wget is from busybox: no > Program 'wget' appears to support flag '--no-config' [14:09:04.188] > Download complete [14:09:04.309] > 071dd51dd448%%2%% [14:09:04.314] > tar --version: tar (GNU tar) 1.34 > Copyright (C) 2021 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later . > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law.Written by John Gilmore and Jay Fenlason. > [14:09:04.391] > mv: cannot move 'code' to 'vscode_server/rocky9-dev-prod-master-myuser/.vscode-server/code-d78a74bcdfad14d5d3b1b782f87255d802b57511': No such file or directory [14:09:04.406] > main: line 337: vscode_server/rocky9-dev-prod-master-myuser/.vscode-server/code-d78a74bcdfad14d5d3b1b782f87255d802b57511: No such file or directory > ERROR: vscode_server/rocky9-dev-prod-master-myuser/.vscode-server/code-d78a74bcdfad14d5d3b1b782f87255d802b57511 don't exist > Trigger local server download > 071dd51dd448:trigger_server_download > artifact==cli-alpine-x64== > destFolder==vscode_server/rocky9-dev-prod-master-myuser/.vscode-server== > destFolder2==/vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz== > 071dd51dd448:trigger_server_download_end > Waiting for client to transfer server archive... > Waiting for vscode_server/rocky9-dev-prod-master-myuser/.vscode-server/vscode-c > cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz.done and vscode-server.tar.gz > z to exist [14:09:04.407] Got request to download on client for {"artifact":"cli-alpine-x64","destPath":"vscode_server/rocky9-dev-prod-master-myuser/.vscode-server/vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz"} [14:09:04.407] server download URL: https://update.code.visualstudio.com/commit:d78a74bcdfad14d5d3b1b782f87255d802b57511/cli-alpine-x64/stable [14:09:04.408] Downloading VS Code server locally... [14:09:04.425] > > [14:09:04.562] Downloaded VS Code server to c:\Users\MYUSER~1\AppData\Local\Temp\433eca01-bd2d-4dd6-839f-16ecf38552b3 [14:09:04.563] Renamed VS Code server to c:\Users\MYUSER~1\AppData\Local\Temp\vscode_server_1728047344562\vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz [14:09:04.563] Preparing to scp to host rocky9-product-master [14:09:04.566] Checking ssh with "C:\Program Files\Zulu\zulu-11\bin\ssh.exe -V" [14:09:04.567] Got error from ssh: spawn C:\Program Files\Zulu\zulu-11\bin\ssh.exe ENOENT [14:09:04.567] Checking ssh with "C:\Program Files\Zulu\zulu-8\bin\ssh.exe -V" [14:09:04.568] Got error from ssh: spawn C:\Program Files\Zulu\zulu-8\bin\ssh.exe ENOENT [14:09:04.568] Checking ssh with "C:\WINDOWS\system32\ssh.exe -V" [14:09:04.569] Got error from ssh: spawn C:\WINDOWS\system32\ssh.exe ENOENT [14:09:04.569] Checking ssh with "C:\WINDOWS\ssh.exe -V" [14:09:04.570] Got error from ssh: spawn C:\WINDOWS\ssh.exe ENOENT [14:09:04.570] Checking ssh with "C:\WINDOWS\System32\Wbem\ssh.exe -V" [14:09:04.571] Got error from ssh: spawn C:\WINDOWS\System32\Wbem\ssh.exe ENOENT [14:09:04.571] Checking ssh with "C:\WINDOWS\System32\WindowsPowerShell\v1.0\ssh.exe -V" [14:09:04.572] Got error from ssh: spawn C:\WINDOWS\System32\WindowsPowerShell\v1.0\ssh.exe ENOENT [14:09:04.572] Checking ssh with "C:\WINDOWS\System32\OpenSSH\ssh.exe -V" [14:09:04.601] > OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2 [14:09:04.603] Expecting scp to be next to ssh at C:\WINDOWS\System32\OpenSSH\scp.exe [14:09:04.603] Testing scp with "C:\WINDOWS\System32\OpenSSH\scp.exe" [14:09:04.639] scp exited with code: 1 [14:09:04.639] Got stderr from scp: usage: scp [-346ABCOpqRrsTv] [-c cipher] [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-J destination] [-l limit] [-o ssh_option] [-P port] [-S program] [-X sftp_option] source ... target [14:09:04.640] Copying file to remote with "C:\WINDOWS\System32\OpenSSH\scp.exe" "vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz" "vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz.done" "rocky9-product-master":"vscode_server/rocky9-dev-prod-master-myuser/.vscode-server" [14:09:04.640] Using cwd: file:///c%3A/Users/MYUSER~1/AppData/Local/Temp/vscode_server_1728047344562 [14:09:04.640] Terminal shell path: C:\WINDOWS\System32\cmd.exe [14:09:04.825] > [?9001h[?1004h [14:09:05.739] > vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz 0% 0 0.0KB/s --:-- ETA]0;C:\WINDOWS\System32\cmd.exe [14:09:06.749] > vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz 92% 7936KB 7.6MB/s 00:00 ETA [14:09:06.824] > vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz 100% 8578KB 7.7MB/s 00:01 [14:09:06.839] > [14:09:07.194] > vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz.done 0% 0 0.0KB/s --:-- ETA [14:09:07.286] > vscode-cli-d78a74bcdfad14d5d3b1b782f87255d802b57511.tar.gz.done 100% 9 0.1KB/s 00:00 [14:09:07.305] > [14:09:08.759] "Copy server to host" terminal command done ```

Anything else?

This used to work quite a while back (I'm not sure what version it stopped working, unfortunately -- probably ~8-12 months).

Then there was an update that caused this issue. I was able to workaround that with "remote.SSH.useExecServer": false, although that also hangs after copying a new version of the server for the first time, but after closing the connection and reconnecting it appears to function correctly.

With VSCode 1.94.0 (I was previously on 1.93.1), the reconnecting no longer appears to work on CentOS 7, with "remote.SSH.useExecServer": false, and it just copies the server and hangs each time I connect, so I hoped I'd be able to get the default use-case with the exec server enabled working again. Another downside of not using the exec server on all OSs I tried is that I had to type in my ssh key password twice each time I connected (after VSCode had already made a successful connection using the ssh agent, from looking at the logs). I'm not sure why that is, but that would be a different issue and I'm more interested in getting the exec server to function, if possible.

moritzwilksch commented 1 month ago

maybe related to #10344?

arghness commented 1 month ago

maybe related to #10344?

Perhaps, but I don't think so -- these hosts all have internet and the log looks very different.

Hjhj789 commented 1 month ago

Yess

joshspicer commented 1 month ago

I'm seeing 

[14:09:04.391] > mv: cannot move 'code' to 'vscode_server/rocky9-dev-prod-master-myuser/.vscode-server/code-d78a74bcdfad14d5d3b1b782f87255d802b57511': No such file or directory

which makes me think that your provided path may be the issue here. I see there's no leading / in your provided Server Install Path, could you try changing that path to include a leading / (or omit that setting entirely for testing).

Note that the /vscode_server/rocky9-dev-prod-master-myuser directory must already exist on the remote and have the appropriate permissions/ownership for the user you're connecting as.

If this is the root cause, we aren't providing a helpful error message for you to diagnose. I'll add this issue to https://github.com/microsoft/vscode-remote-release/issues/10249 to improve the error we emit

Please let me know if that helps (providing new logs if not 🙂)

arghness commented 1 month ago

I'm seeing

[14:09:04.391] > mv: cannot move 'code' to 'vscode_server/rocky9-dev-prod-master-myuser/.vscode-server/code-d78a74bcdfad14d5d3b1b782f87255d802b57511': No such file or directory

which makes me think that your provided path may be the issue here. I see there's no leading / in your provided Server Install Path, could you try changing that path to include a leading / (or omit that setting entirely for testing).

Note that the /vscode_server/rocky9-dev-prod-master-myuser directory must already exist on the remote and have the appropriate permissions/ownership for the user you're connecting as.

If this is the root cause, we aren't providing a helpful error message for you to diagnose. I'll add this issue to #10249 to improve the error we emit

Please let me know if that helps (providing new logs if not 🙂)

Hi, thanks, that was the issue!

I was trying to install in what was effectively ~/vscode_server/ so I didn't want a / prefix. The issue probably occurred after #9474 became an issue, as I previously had $HOME and $HOSTNAME but they stopped expanding correctly. It still seemed to actually do the download to the correct directory using a relative path, but only actually worked with "remote.SSH.useExecServer": false (after a reconnect after doing the download). There must be a difference with how those 2 modes handle relative paths.

Anyway, it now appears that the remote extension can expand $HOME and $HOSTNAME again in remote.SSH.serverInstallPath (although #9474 wasn't marked as fixed), so using "$HOME/vscode_server/$HOSTNAME" works without disabling useExecServer.

Thank you for helping!

joshspicer commented 1 month ago

Glad you're unblocked, and thanks for the added details!