microsoft / vscode-remote-release

Visual Studio Code Remote Development: Open any folder in WSL, in a Docker container, or on a remote machine using SSH and take advantage of VS Code's full feature set.
https://aka.ms/vscode-remote
Other
3.55k stars 263 forks source link

Investigate using local server on remoteListenOnSocket #6086

Open jasperhyp opened 2 years ago

jasperhyp commented 2 years ago

I was trying to get onto the second layer of an HPC after SSH-ing with some Slurm command (srun), as was generally described in #1722. I tried the newest RemoteCommand from the Pre-release version of Remote SSH (following instructions in this issue), but it seems not passing the RemoteCommand when connecting.

Logs ``` [11:47:37.895] Log Level: 2 [11:47:37.897] remote-ssh@0.71.2021121615 [11:47:37.897] win32 x64 [11:47:37.915] SSH Resolver called for "ssh-remote+some.host.name", attempt 1 [11:47:37.915] "remote.SSH.useLocalServer": false [11:47:37.916] "remote.SSH.showLoginTerminal": false [11:47:37.916] "remote.SSH.remotePlatform": {"instance-1":"linux","some.host.name":"linux"} [11:47:37.916] "remote.SSH.path": undefined [11:47:37.916] "remote.SSH.configFile": undefined [11:47:37.916] "remote.SSH.useFlock": true [11:47:37.916] "remote.SSH.lockfilesInTmp": false [11:47:37.916] "remote.SSH.localServerDownload": auto [11:47:37.916] "remote.SSH.remoteServerListenOnSocket": true [11:47:37.917] "remote.SSH.showLoginTerminal": false [11:47:37.917] "remote.SSH.defaultExtensions": [] [11:47:37.917] "remote.SSH.loglevel": 2 [11:47:37.917] "remote.SSH.enableDynamicForwarding": true [11:47:37.917] "remote.SSH.enableRemoteCommand": true [11:47:37.917] "remote.SSH.serverPickPortsFromRange": {} [11:47:37.917] "remote.SSH.serverInstallPath": {} [11:47:37.918] SSH Resolver called for host: some.host.name [11:47:37.918] Setting up SSH remote "some.host.name" [11:47:37.922] Using commit id "899d46d82c4c95423fb7e10e68eba52050e30ba3" and quality "stable" for server [11:47:37.925] Install and start server if needed [11:47:37.927] Checking ssh with "ssh -V" [11:47:37.961] > OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 (HERE) [11:47:37.964] Running script with connection command: ssh -T -D 8501 -o RemoteCommand=none "some.host.name" bash [11:47:37.966] Terminal shell path: C:\Windows\System32\cmd.exe [11:47:38.158] > ]0;C:\Windows\System32\cmd.exe [11:47:38.158] Got some output, clearing connection timeout [11:47:38.409] > P [11:47:38.446] > Password: [11:47:38.447] Showing password prompt [11:47:43.428] Got password response [11:47:43.429] "install" wrote data to terminal: "*******************" [11:47:43.446] > [11:47:58.917] > 91a30fe8ff8e: running [11:47:59.003] > Acquiring lock on /home/yeh803/.vscode-server/bin/899d46d82c4c95423fb7e10e68eba5 > 2050e30ba3/vscode-remote-lock.yeh803.899d46d82c4c95423fb7e10e68eba52050e30ba3 [11:47:59.041] > Found existing installation at /home/yeh803/.vscode-server/bin/899d46d82c4c95423 > fb7e10e68eba52050e30ba3... [11:47:59.075] > Checking /home/yeh803/.vscode-server/.899d46d82c4c95423fb7e10e68eba52050e30ba3.l > og and /home/yeh803/.vscode-server/.899d46d82c4c95423fb7e10e68eba52050e30ba3.pid > for a running server [11:47:59.084] > Looking for server with pid: 31076 [11:47:59.173] > Starting server with command... /home/yeh803/.vscode-server/bin/899d46d82c4c9542 > 3fb7e10e68eba52050e30ba3/server.sh --start-server --host=127.0.0.1 --enable-remo > te-auto-shutdown --socket-path=/run/user/184812/vscode-ssh-remote-server-sock- > 1640018879 &> "/home/yeh803/.vscode-server/.899d46d82c4c95423fb7e10e68eba52050e > 30ba3.log" < /dev/null > printenv: > MANPATH=/n/app/lmod/lmod/share/man:: > XDG_SESSION_ID=20830 > rvm_bin_path=/usr/local/rvm/bin > GEM_HOME=/usr/local/rvm/gems/ruby-2.4.9 > SHELL=/bin/bash > IRBRC=/usr/local/rvm/rubies/ruby-2.4.9/.irbrc > MODULEPATH_ROOT=/n/app/lmod/lmod/modulefiles > SSH_CLIENT=73.61.9.36 9459 22 > CONDA_SHLVL=1 > CONDA_PROMPT_MODIFIER=(base) > LMOD_PKG=/n/app/lmod/lmod [11:47:59.181] > > MY_RUBY_HOME=/usr/local/rvm/rubies/ruby-2.4.9 > LMOD_VERSION=7.5.17 > QT_GRAPHICSSYSTEM_CHECKED=1 > USER=yeh803 > LMOD_sys=Linux > CONDA_EXE=/n/app/conda2/bin/conda > rvm_path=/usr/local/rvm > VSCODE_AGENT_FOLDER=/home/yeh803/.vscode-server > _CE_CONDA= > rvm_prefix=/usr/local > MAIL=/var/mail/yeh803 > PATH=/n/app/conda2/bin:/n/app/conda2/condabin:/usr/local/rvm/gems/ruby-2.4.9 > /bin:/usr/local/rvm/gems/ruby-2.4.9@global/bin:/usr/local/rvm/rubies/ruby-2.4.9/ > bin:/n/cluster/bin:/opt/singularity/bin:/usr/local/bin:/usr/bin:/opt/puppetlabs/ > bin:/usr/local/rvm/bin > SLURM_CONF=/etc/slurm/slurm.conf > CONDA_PREFIX=/n/app/conda2 > LMOD_SETTARG_CMD=: > PWD=/home/yeh803 > LANG=en_US.UTF-8 > MODULEPATH=/n/app/lmod/lmod/modulefiles/Linux:/n/app/lmod/lmod/modulefiles/C > ore > TZ=America/New_York > LMOD_CMD=/n/app/lmod/lmod/libexec/lmod > _CE_M= > HMS_CLUSTER=o2 > KRB5CCNAME=FILE:/tmp/krb5cc_184812_zPUZlA > rvm_version=1.29.10 (latest) > SHLVL=3 > HOME=/home/yeh803 > BASH_ENV=/n/app/lmod/lmod/init/bash > SSH_ORIGINAL_COMMAND=bash > CONDA_PYTHON_EXE=/n/app/conda2/bin/python > LOGNAME=yeh803 > GEM_PATH=/usr/local/rvm/gems/ruby-2.4.9:/usr/local/rvm/gems/ruby-2.4.9@globa > l > SSH_CONNECTION=73.61.9.36 9459 134.174.159.26 22 > MODULESHOME=/n/app/lmod/lmod > CONDA_DEFAULT_ENV=base > LESSOPEN=||/usr/bin/lesspipe.sh %s > SHARED_DATABASES=/n/groups/shared_databases > LMOD_FULL_SETTARG_SUPPORT=no > XDG_RUNTIME_DIR=/run/user/184812 > LMOD_DIR=/n/app/lmod/lmod/libexec > RUBY_VERSION=ruby-2.4.9 > BASH_FUNC_module()=() { eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTAR > G_CMD:-:} -s sh) > } > BASH_FUNC_ml()=() { eval $($LMOD_DIR/ml_cmd "$@") > } > _=/usr/bin/printenv [11:47:59.188] > [11:47:59.207] > Spawned remote server: 104641 > Waiting for server log... [11:47:59.223] > [11:47:59.274] > Waiting for server log... [11:47:59.374] > Waiting for server log... [11:47:59.471] > Waiting for server log... [11:47:59.571] > Waiting for server log... [11:47:59.673] > Waiting for server log... [11:47:59.773] > Waiting for server log... [11:47:59.874] > Waiting for server log... [11:47:59.972] > Waiting for server log... [11:48:00.070] > Waiting for server log... [11:48:00.172] > Waiting for server log... [11:48:00.273] > Waiting for server log... [11:48:00.373] > Waiting for server log... [11:48:00.474] > Waiting for server log... [11:48:00.575] > Waiting for server log... [11:48:00.673] > Waiting for server log... [11:48:00.709] > Waiting for server log... [11:48:00.771] > Waiting for server log... [11:48:00.881] > > * > * Reminder: You may only use this software with Visual Studio family products, > * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057) > * > [11:48:00.906] > [11:48:00.974] > 91a30fe8ff8e: start > SSH_AUTH_SOCK==== > DISPLAY==== > webUiAccessToken==== > listeningOn==/run/user/184812/vscode-ssh-remote-server-sock-1640018879== > osReleaseId==centos== > arch==x86_64== > tmpDir==/run/user/184812== > platform==linux== > unpackResult==== > didLocalDownload==0== > downloadTime==== > installTime==== > extInstallTime==== > serverStartTime==1695== > connectionToken==aa11111a-1aa1-111a-a111-11111a111a1a== > 91a30fe8ff8e: end [11:48:00.976] Received install output: SSH_AUTH_SOCK==== DISPLAY==== webUiAccessToken==== listeningOn==/run/user/184812/vscode-ssh-remote-server-sock-1640018879== osReleaseId==centos== arch==x86_64== tmpDir==/run/user/184812== platform==linux== unpackResult==== didLocalDownload==0== downloadTime==== installTime==== extInstallTime==== serverStartTime==1695== connectionToken==aa11111a-1aa1-111a-a111-11111a111a1a== [11:48:00.977] Remote server is listening on /run/user/184812/vscode-ssh-remote-server-sock-1640018879 [11:48:00.977] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"socketPath":"/run/user/184812/vscode-ssh-remote-server-sock-1640018879"},"osReleaseId":"centos","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/184812","platform":"linux","connectionToken":"aa11111a-1aa1-111a-a111-11111a111a1a"},"serverStartTime":1695,"installUnpackCode":""} (HERE) [11:48:00.979] Running script with connection command: ssh -T -L 127.0.0.1:8507:/run/user/184812/vscode-ssh-remote-server-sock-1640018879 -o RemoteCommand=none "some.host.name" bash [11:48:00.980] Spawning tunnel with: type "C:\Users\Yep\AppData\Local\Temp\vscode-linux-multi-line-command-some.host.name-223392234.sh" | ssh -T -L 127.0.0.1:8507:/run/user/184812/vscode-ssh-remote-server-sock-1640018879 -o RemoteCommand=none "some.host.name" bash [11:48:00.981] Terminal shell path: C:\Windows\System32\cmd.exe [11:48:00.995] > [11:48:01.171] > ]0;C:\Windows\System32\cmd.exe [11:48:01.538] > P [11:48:01.822] > Password: [11:48:01.823] Showing password prompt [11:48:06.095] Got password response [11:48:06.095] "SSH Tunnel" wrote data to terminal: "*******************" [11:48:06.127] > [11:48:14.043] > SSH_AUTH_SOCK= > Connected to SSH Host - Please do not close this terminal [11:48:14.044] Spawned SSH tunnel between local port 8507 and remote target /run/user/184812/vscode-ssh-remote-server-sock-1640018879 [11:48:14.044] Waiting for ssh tunnel to be ready [11:48:14.045] Tunneled /run/user/184812/vscode-ssh-remote-server-sock-1640018879 to local port 8507 [11:48:14.045] Resolved "ssh-remote+some.host.name" to "127.0.0.1:8507" [11:48:14.048] ------ ```

Steps to Reproduce:

  1. Add "remote.SSH.enableRemoteCommand": true, "remote.SSH.useLocalServer": true, "remote.SSH.remoteServerListenOnSocket": true to settings.json. "remote.SSH.remoteServerListenOnSocket": true is a must because otherwise would result in Failed to set up socket for dynamic port forward to remote port 45824: Socket closed.

  2. SSH to some.host.name with the following ssh config:

    Host some.host.name
    User some.user.name
    // ForwardAgent yes  // Seems this does not matter
    RemoteCommand srun -t 0-12:00 --pty -p gpu --gres=gpu:1 /bin/bash

Does this issue occur when you try this locally?: It's a connection issue.

tanhakabir commented 2 years ago

Hm and when you connect to this same host through PowerShell do you see your command in RemoteCommand being passed as expected?

jasperhyp commented 2 years ago

Hm and when you connect to this same host through PowerShell do you see your command in RemoteCommand being passed as expected?

Yes, when I use PowerShell with ssh host.name.in.the.ssh.config, this RemoteCommand is passed, as shown in the snapshot below.

image

Never mind the error message. It just showed the remote command has been passed.

jasperhyp commented 2 years ago

Well... maybe this error message is a problem even if the RemoteCommand can be passed in VSCode-Remote, since I am also getting into this error using ssh something 'srun something' command in MobaXterm (but np if I run srun something after ssh something to the login node).

However, it stays true that RemoteCommand (even as simple as ls) is not passed in VSCode, while both PowerShell and Moba pass it:

image

image

image
tanhakabir commented 2 years ago

Oh I missed this earlier. I see in your logs you have [11:47:37.915] "remote.SSH.useLocalServer": false

You need to have useLocalServer enabled to use RemoteCommand. That's why you see in the connection command RemoteCommand=none

jasperhyp commented 2 years ago

Oh I missed this earlier. I see in your logs you have [11:47:37.915] "remote.SSH.useLocalServer": false

You need to have useLocalServer enabled to use RemoteCommand. That's why you see in the connection command RemoteCommand=none

Oh looks like this parameter is set to false in ssh. But I did set the parameter to true in settings.json, and the box before Remote.SSH: Use Local Server is shown to be marked:

    "remote.SSH.useLocalServer": true,
    "remote.SSH.enableRemoteCommand": true,
    "terminal.integrated.inheritEnv": false,
    "remote.SSH.remoteServerListenOnSocket": true,
    "window.zoomLevel": 1,

image

tanhakabir commented 2 years ago

Try setting "remote.SSH.remoteServerListenOnSocket" to false.

They can't work in conjunction.

jasperhyp commented 2 years ago
  1. Failed to set up socket for dynamic port forward to remote port 45824: Socket closed

Thank you, I actually tried set "remote.SSH.remoteServerListenOnSocket": false and "remote.SSH.useLocalServer": true at the time I filed this issue, but it yielded an error Failed to set up socket for dynamic port forward to remote port XXXX: Socket closed. It's still the case now.

tanhakabir commented 2 years ago

Oh I see, hm that's unfortunate. Right now we can only support RemoteCommand when local server is enabled but for some machines that method doesn't work and remoteServerListenOnSocket is needed like your machine seems to. I'm not sure if there's a resolution for this.

jasperhyp commented 2 years ago

Oh I see, hm that's unfortunate. Right now we can only support RemoteCommand when local server is enabled but for some machines that method doesn't work and remoteServerListenOnSocket is needed like your machine seems to. I'm not sure if there's a resolution for this.

Got it, thanks! I tried again today and it just doesn't work. Maybe I'll just use VS Code to view the code but not run anything.

tanhakabir commented 2 years ago

Sorry about that! Do you need RemoteCommand to run/compile your code?

github-actions[bot] commented 2 years ago

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

jasperhyp commented 2 years ago

Sorry about that! Do you need RemoteCommand to run/compile your code?

Thanks for the follow-up. Have been out of work for a few days. I am mainly using Python so usually compilation is not needed. I just sometimes want to use .ipynb files to test some code. My current workflow is to use VS Code's Remote SSH for viewing & editing (because I love your great add-ons), and use Jupyter notebook & port-forwarding to test small code. That works despite a bit more windows.

tanhakabir commented 2 years ago

use Jupyter notebook & port-forwarding to test small code

Is this done locally on your machine or on a remote machine? I believe everything you do locally should be able to be done in your remote if you wish it have it running there. Maybe just not as a RemoteCommand that runs on connection to the remote.

jasperhyp commented 2 years ago

use Jupyter notebook & port-forwarding to test small code

Is this done locally on your machine or on a remote machine? I believe everything you do locally should be able to be done in your remote if you wish it have it running there. Maybe just not as a RemoteCommand that runs on connection to the remote.

I meant to say "open Jupyter Notebook on the remote machine and port forward to local machine with browser". I cannot do this with Remote SSH because I need to run RemoteCommand to get to the second (computing) layer of the HPC to run .ipynb files (since they are too resource-intensive for a login layer).

tanhakabir commented 2 years ago

Ahh I see. Thanks for explaining! Sorry about that! I'll explore alternatives to localServer with remoteListenOnSocket is needed.

ilyasdc commented 2 years ago

I face the same issue. I need to use slurm for ressource allocation because the login nodes are slow (they are login nodes after all) and cannot handle vscode server properly (ie most features are significanly slowed-down).

Standard ssh with port forwarding directly to the compute node doesn't works (ssh -L) but I can reverse forward from the compute to the login node (e.g. I can forward tensorboard running on the compute node)

my .ssh/config content (I need to use a jump node to get access to the login node)

Host zzz_vscode
     HostName zzz
     User yyy
     ProxyCommand XXX
     RequestTTY yes  
     RemoteCommand srun -N1  --account=xxx --cpus-per-task=1 --pty bash
Vscode SSH extension logs ``` [14:40:05.498] Log Level: 1 [14:40:05.499] remote-ssh@0.73.2022020215 [14:40:05.499] darwin arm64 [14:40:05.500] SSH Resolver called for "ssh-remote+zzz_vscode", attempt 1 [14:40:05.500] "remote.SSH.useLocalServer": true [14:40:05.500] "remote.SSH.path": undefined [14:40:05.500] "remote.SSH.configFile": undefined [14:40:05.500] "remote.SSH.useFlock": true [14:40:05.500] "remote.SSH.lockfilesInTmp": false [14:40:05.501] "remote.SSH.localServerDownload": auto [14:40:05.501] "remote.SSH.remoteServerListenOnSocket": false [14:40:05.501] "remote.SSH.showLoginTerminal": false [14:40:05.501] "remote.SSH.defaultExtensions": [] [14:40:05.502] "remote.SSH.loglevel": 1 [14:40:05.502] "remote.SSH.enableDynamicForwarding": true [14:40:05.502] "remote.SSH.enableRemoteCommand": true [14:40:05.502] "remote.SSH.serverPickPortsFromRange": {} [14:40:05.502] "remote.SSH.serverInstallPath": {} [14:40:05.508] SSH Resolver called for host: zzz_vscode [14:40:05.508] Setting up SSH remote "zzz_vscode" [14:40:05.510] Acquiring local install lock: /var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-remote-ssh-db60dc3f-install.lock [14:40:05.511] Looking for existing server data file at /Users/yyy/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-db60dc3f-62de8dc3f78b8bef1eb62deeb093164a8d76c254-0.73.2022020215/data.json [14:40:05.511] Found local server running: {"remoteListeningOn":{"port":33051},"osReleaseId":"centos","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/8675","platform":"linux","connectionToken":"11aaaa11-1111-11aa-a111-11aa11111a11","pid":82322,"ipcHandlePath":"/var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-d85a4b01a2eb3c8ceee690abf0c737630364345c.sock","socksPort":50839,"startupTime":1644143915576} [14:40:05.513] Server delay-shutdown request failed: connect ECONNREFUSED /var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-d85a4b01a2eb3c8ceee690abf0c737630364345c.sock [14:40:05.513] Existing server's health check failed, will start a new one [14:40:05.513] Using commit id "62de8dc3f78b8bef1eb62deeb093164a8d76c254" and quality "insider" for server [14:40:05.515] Install and start server if needed [14:40:05.518] PATH: /Users/yyy/.jenv/shims:/Users/yyy/.jenv/bin:/opt/homebrew/opt/rclone-mac/libexec/rclone:/Users/yyy/.miniconda3/condabin:/opt/local/bin:/opt/local/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion Tech Preview.app/Contents/Public:/usr/local/MacGPG2/bin:/opt/X11/bin:/Library/Apple/usr/bin:/opt/homebrew/opt/fzf/bin [14:40:05.518] Checking ssh with "ssh -V" [14:40:05.527] > OpenSSH_8.6p1, LibreSSL 2.8.3 [14:40:05.528] askpass server listening on /var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-6b4f514209bbb52a2c2afe1e255de03f4a726354.sock [14:40:05.528] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-646a156a10d8d7b8547059b25e7c52b9a475015b.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","63353","-o","ConnectTimeout=15","zzz_vscode"],"serverDataFolderName":".vscode-server-insiders","dataFilePath":"/Users/yyy/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-db60dc3f-62de8dc3f78b8bef1eb62deeb093164a8d76c254-0.73.2022020215/data.json"} [14:40:05.528] Local server env: {"DISPLAY":"/private/tmp/com.apple.launchd.j9qwA5HaxM/org.xquartz:0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/yyy/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.73.2022020215/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code - Insiders.app/Contents/MacOS/Electron","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/Users/yyy/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.73.2022020215/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-6b4f514209bbb52a2c2afe1e255de03f4a726354.sock"} [14:40:05.529] Spawned 89305 [14:40:05.579] > local-server-1> Spawned ssh, pid=89315 [14:40:05.581] stderr> OpenSSH_8.6p1, LibreSSL 2.8.3 [14:40:05.581] stderr> debug1: Reading configuration data /Users/yyy/.ssh/config [14:40:05.581] stderr> debug1: /Users/yyy/.ssh/config line 6: Applying options for * [14:40:05.581] stderr> debug1: /Users/yyy/.ssh/config line 41: Applying options for zzz_vscode [14:40:05.581] stderr> debug1: Reading configuration data /etc/ssh/ssh_config [14:40:05.581] stderr> debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files [14:40:05.581] stderr> debug1: /etc/ssh/ssh_config line 54: Applying options for * [14:40:05.582] stderr> debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling [14:40:05.582] stderr> debug1: auto-mux: Trying existing master [14:40:05.582] stderr> debug1: Requesting forwarding of dynamic forward LOCALHOST:63353 -> * [14:40:05.592] stderr> debug1: mux_client_request_session: master session id: 12 [14:40:06.196] stderr> manpath: can't set the locale; make sure $LC_* and $LANG are correct [14:40:06.260] stderr> srun: error: ioctl(TIOCGWINSZ): Inappropriate ioctl for device [14:40:06.260] stderr> srun: error: Not using a pseudo-terminal, disregarding --pty option [14:40:06.421] > ready: b0826ae25f94 [14:40:06.434] > Linux 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 [14:40:06.434] Platform: linux [14:40:06.484] > b0826ae25f94: running [14:40:06.524] > Acquiring lock on /home/yyy/.vscode-server-insiders/bin/62de8dc3f78b8bef1eb62deeb093164a8d76c254/vscode-remote-lock.yyy.62de8dc3f78b8bef1eb62deeb093164a8d76c254 [14:40:06.530] > Found existing installation at /home/yyy/.vscode-server-insiders/bin/62de8dc3f78b8bef1eb62deeb093164a8d76c254... > Checking /home/yyy/.vscode-server-insiders/.62de8dc3f78b8bef1eb62deeb093164a8d76c254.log and /home/yyy/.vscode-server-insiders/.62de8dc3f78b8bef1eb62deeb093164a8d76c254.pid for a running server [14:40:06.577] > Looking for server with pid: 10841 [14:40:06.627] > Starting server with command... /home/yyy/.vscode-server-insiders/bin/62de8dc3f78b8bef1eb62deeb093164a8d76c254/bin/code-server-insiders --start-server --host=127.0.0.1 --accept-server-license-terms --enable-remote-auto-shutdown --port=0 &> "/home/yyy/.vscode-server-insiders/.62de8dc3f78b8bef1eb62deeb093164a8d76c254.log" < /dev/null > printenv: [14:40:06.631] > NVM_INC=/home/yyy/.nvm/versions/node/v14.18.1/include/node > SLURM_NODELIST=zzz-n5 > SLURM_JOB_NAME=bash > XDG_SESSION_ID=549 > SLURM_TOPOLOGY_ADDR=zzz-n5 > SLURMD_NODENAME=zzz-n5 > SELINUX_ROLE_REQUESTED= > SLURM_PRIO_PROCESS=0 > NVM_CD_FLAGS= > SLURM_SRUN_COMM_PORT=62456 > SHELL=/bin/bash > SLURM_JOB_QOS=normal > SSH_CLIENT=213.135.51.79 41156 22 > SLURM_TOPOLOGY_ADDR_PATTERN=node > TMPDIR=/tmp > CONDA_SHLVL=0 > SELINUX_USE_CURRENT_RANGE= > SLURM_CPU_BIND_VERBOSE=quiet > SLURM_CPU_BIND_LIST=0x000000002 > HISTFILESIZE=10000 > NVM_DIR=/home/yyy/.nvm > USER=yyy > SLURM_NNODES=1 > CONDA_EXE=/home/yyy/.miniconda/bin/conda > SLURM_STEP_NUM_NODES=1 > SRUN_DEBUG=3 > SLURM_JOBID=51884 > SLURM_NTASKS=1 > SLURM_LAUNCH_NODE_IPADDR=172.24.40.5 > SLURM_STEP_ID=0 > VSCODE_AGENT_FOLDER=/home/yyy/.vscode-server-insiders > _CE_CONDA= > MAIL=/var/mail/yyy > PATH=/home/yyy/.nvm/versions/node/v14.18.1/bin:/home/yyy/.miniconda/condabin:/usr/local/bin:/usr/bin:/apps/icm-tools/bin > SLURM_TASKS_PER_NODE=1 > SLURM_STEP_LAUNCHER_PORT=62456 > SLURM_CONF=/usr/local/etc/slurm/slurm.conf > SLURM_WORKING_CLUSTER=zzz:zzz-mgmt2.icm.edu.pl:6817:8960:101 > SLURM_CPUS_PER_TASK=1 > SLURM_JOB_ID=51884 > PWD=/lu/tetyda/home/yyy > SLURM_STEPID=0 > SLURM_JOB_USER=yyy > SLURM_SRUN_COMM_HOST=172.24.40.5 > SLURM_CPU_BIND_TYPE=mask_cpu: > LANG=en-US > MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles > SLURM_UMASK=0022 > LOADEDMODULES= > SLURM_JOB_UID=8675 > SLURM_NODEID=0 > SELINUX_LEVEL_REQUESTED= > SLURM_SUBMIT_DIR=/lu/tetyda/home/yyy > SLURM_STEP_RESV_PORTS=12371-12372 > SLURM_NPROCS=1 > SLURM_TASK_PID=12591 > SLURM_CPUS_ON_NODE=1 > HISTCONTROL=ignorespace:erasedups > _CE_M= > SLURM_PROCID=0 > SLURM_JOB_NODELIST=zzz-n5 > SHLVL=3 > HOME=/home/yyy > SLURM_LOCALID=0 > SLURM_CLUSTER_NAME=zzz > SLURM_JOB_CPUS_PER_NODE=1 > SLURM_JOB_GID=2000 > SLURM_SUBMIT_HOST=zzz.icm.edu.pl > SLURM_GTIDS=0 > SLURM_JOB_PARTITION=gpu > CONDA_PYTHON_EXE=/home/yyy/.miniconda/bin/python > LOGNAME=yyy [14:40:06.633] > SLURM_STEP_NUM_TASKS=1 > SSH_CONNECTION=213.135.51.79 41156 172.24.40.5 22 > SLURM_JOB_ACCOUNT=gr85-0 > MODULESHOME=/usr/share/Modules > SLURM_JOB_NUM_NODES=1 > NVM_BIN=/home/yyy/.nvm/versions/node/v14.18.1/bin > LESSOPEN=||/usr/bin/lesspipe.sh %s > SLURM_STEP_TASKS_PER_NODE=1 > SLURM_STEP_NODELIST=zzz-n5 > XDG_RUNTIME_DIR=/run/user/8675 > SLURM_CPU_BIND=quiet,mask_cpu:0x000000002 > BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*` > } > _=/usr/bin/printenv [14:40:06.652] > Spawned remote server: 12637 [14:40:06.663] > Waiting for server log... [14:40:06.724] > Waiting for server log... [14:40:06.798] > Waiting for server log... [14:40:06.866] > [14:40:06.875] > * > * Reminder: You may only use this software with Visual Studio family products, > * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057) > * > [14:40:06.885] > b0826ae25f94: start > SSH_AUTH_SOCK==== > DISPLAY==== > webUiAccessToken==== > listeningOn==41925== > osReleaseId==centos== > arch==x86_64== > tmpDir==/run/user/8675== > platform==linux== > unpackResult==== > didLocalDownload==0== > downloadTime==== > installTime==== > extInstallTime==== > serverStartTime==230== > connectionToken==a1a1111a-1111-1a11-1aa1-11a1a11aa11a== > b0826ae25f94: end [14:40:06.885] Received install output: SSH_AUTH_SOCK==== DISPLAY==== webUiAccessToken==== listeningOn==41925== osReleaseId==centos== arch==x86_64== tmpDir==/run/user/8675== platform==linux== unpackResult==== didLocalDownload==0== downloadTime==== installTime==== extInstallTime==== serverStartTime==230== connectionToken==a1a1111a-1111-1a11-1aa1-11a1a11aa11a== [14:40:06.886] Remote server is listening on 41925 [14:40:06.886] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":41925},"osReleaseId":"centos","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/8675","platform":"linux","connectionToken":"a1a1111a-1111-1a11-1aa1-11a1a11aa11a"},"serverStartTime":230,"installUnpackCode":""} [14:40:06.887] Persisting server connection details to /Users/yyy/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-db60dc3f-62de8dc3f78b8bef1eb62deeb093164a8d76c254-0.73.2022020215/data.json [14:40:06.890] Starting forwarding server. localPort 63357 -> socksPort 63353 -> remotePort 41925 [14:40:06.890] Forwarding server listening on 63357 [14:40:06.890] Waiting for ssh tunnel to be ready [14:40:06.891] Tunneled 41925 to local port 63357 [14:40:06.891] Resolved "ssh-remote+zzz_vscode" to "127.0.0.1:63357" [14:40:06.892] Updating terminal environments: {} [14:40:06.892] [Forwarding server 63357] Got connection 0 [14:40:06.898] TELEMETRY: {"eventName":"resolver","properties":{"osReleaseId":"centos","arch":"x86_64","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"1","gotUnrecognizedPrompt":"0","dynamicForwarding":"1","localServer":"1","didLocalDownload":"0","installUnpackCode":"0","outcome":"success"},"measures":{"resolveAttempts":1,"retries":1,"timing.totalResolveTime":1398,"timing.preSshTime":28,"timing.establishSshTime":956,"timing.scriptTime":1358,"timing.serverStartTime":230}} [14:40:06.901] ------ [14:40:06.913] Failed to set up socket for dynamic port forward to remote port 41925: Socket closed. Is the remote port correct? [14:40:06.916] [Forwarding server 63357] Got connection 1 [14:40:06.926] Failed to set up socket for dynamic port forward to remote port 41925: Socket closed. Is the remote port correct? ```
tzom commented 2 years ago

Ahh I see. Thanks for explaining! Sorry about that! I'll explore alternatives to localServer with remoteListenOnSocket is needed.

@tanhakabir did you had a chance to look at it and pinpoint the issue? would be really cool to just be able to use: RemoteCommand srun --pty --gres=gpu:1 /bin/bash and let vscode figure out which ports to forward/listen to, automatically.

schroeni96 commented 2 years ago

Greeting! I just found this Issue while I tried to achieve something similar (Running VSCode on a Slurm Node with singularity, Using RemoteCommand and LocalServer) and I am facing the exact same issue as @ilyasdc. The output of the install procedure looks almost identical, with the final error

Failed to set up socket for dynamic port forward to remote port

Can I somehow help with this Issue, maybe by providing Logfiles or similar? This feature would greatly benefit the research at my facility, so I would be very glad to help.

adinriv commented 2 years ago
  1. Failed to set up socket for dynamic port forward to remote port 45824: Socket closed

Thank you, I actually tried set "remote.SSH.remoteServerListenOnSocket": false and "remote.SSH.useLocalServer": true at the time I filed this issue, but it yielded an error Failed to set up socket for dynamic port forward to remote port XXXX: Socket closed. It's still the case now.

I'm seeing similar forwarding errors when using the RemoteCommand when trying to run the allocation through the command. So, same behavior as the one reported by @jasperhyp and @ilyasdc .

I can, however, execute commands directly on the login node.

It seems that something with the jump on the resource allocation (srun or salloc) messes up the setup from VSCode.

Happy to debug in case you need some logs.

jasperhyp commented 2 years ago

Workaround from a very clever friend (credits to Gwen, whose GitHub id I don't know about):

For those who only need to srun to the compute node, try first srun a compute node in server using anything else and then use ProxyJump command in config.ssh, e.g.

Host login-node
    HostName login-node.edu
    User hello

Host compute-node
    HostName %h
    ProxyJump login-node
    User hello

The assumption here is that you can use ssh compute-node in your server once you log onto the login node. Works for my case.

Also see here

ilyasdc commented 2 years ago

Workaround from a very clever friend (credits to Gwen, whose GitHub id I don't know about):

For those who only need to srun to the compute node, try first srun a compute node in server using anything else and then use ProxyJump command in config.ssh, e.g.

Host login-node
    HostName login-node.edu
    User hello

Host compute-node
    HostName %h
    ProxyJump login-node
    User hello

The assumption here is that you can use ssh compute-node in your server once you log onto the login node. Works for my case.

Also see here

If only it was as imple as that ... I cannot directly ssh to the compute node !

saiden89 commented 2 years ago

I'm also facing the same usecase as @ilyasdc, but using a TORQUE instead of SLURM. Of course, the workaround proposed by @jasperhyp does not really work on batch systems as SSHing into compute nodes is forbidden. To me, it seems that the extension is ignoring the RequestTTY yes because in a normal terminal it works just fine. My config is:

Host login-interactive
  HostName ssh.login-node.edu
  User myuser
  RemoteCommand  qsub -I -W group_list=GROUP -A GROUP -l nodes=1:ppn=1,mem=1g,walltime=01:00:00
  RequestTTY force

Excerpt of output:

Authenticated to ssh.login-node.edu ([IP]:22).
qsub:   standard input and output must be a terminal for 
    interactive job submission
tzom commented 1 year ago

hi everyone :) any news on this issue? would be really helpful to have RemoteCommand+Slurm working together...

PascalIversen commented 1 year ago

My team and I are also desperately waiting for a solution to this. Any updates on the progress? Thanks!

alex-necula commented 1 year ago

Bumping this up

eleanorjboyd commented 1 year ago

Hello! Could the people still interest in the thread outline their exact use case for this need. This would be a complicated fix so we are trying to determine if there are any solutions this problem that might be more successful and quicker. Thanks!

tzom commented 1 year ago

Hi @eleanorjboyd ,

the original post describes it best (first post in this issue) https://github.com/microsoft/vscode-remote-release/issues/6086#issue-1085195764

The overall goal is to run the vscode-server on a compute node, e.g. in a computer cluster. Specifically, with RemoteCommand in the ssh config file we try to allocate a resource (via slurm or any other job scheduler) and let vscode-server run on the allocated node:

RemoteCommand srun --mem=16G -c=1 --time=08:00:00 --pty bash

This is crucial because in many compute clusters it is not allowed to run something like vscode-server on the head node itself. So RemoteCommand would solve this as it does run the slurm command first and then starts the vscode-server on the allocated resource. However, RemoteCommand does not exactly behave like it is supposed to. This issue #6086 narrows it down to the settings ("remoteServerListenOnSocket", "useLocalServer", etc.) that may interfere with RemoteCommand. Also, this issue is related to this issue #1722

hope it helps. Looking forward to having RemoteCommand properly working in combination with a compute cluster.

ekungurov commented 1 year ago

Hi @eleanorjboyd , I believe there still should be interest.

The proposed solution is to run code-server on a remote machine we need to connect. In our case, it means scheduling a job via cluster management software (e.g. slurm) to run code-server binary on a computational node, and then get the stdout from code-server with connection details (pin code). Then use VSCode "Remote - Tunnels" plugin to connect to that code-server instance.

Downside of the proposed solution is: It uses Microsoft infrastructure to tunnel the traffic between Remote Tunnel plugin and code-server instance. If everything is in the same local net, this shouldn't be needed.

With "Remote - SSH" plugin the issue is the following. It allows to customize RemoteCommand, and we can use a customized RemoteCommand to run 'Code' on a computational node (instead of the node we established ssh connection to) and also to set up our own transparent tunneling between the computational node and the slurm control node, so that the 'Code' running on a computational node could be contacted via socket on a control node. However, we don't have a way for the Visual Studio Code (which is being run on a local PC) to use that particular socket instead of the one it thinks it have to use.

Hope I described the use case. If it's still not very clear, feel free to engage someone technical from your company (I'm sorry if it sounds rude, I just don't want this comment to be sort of useless or lost, and don't know how to express it better in words).

roblourens commented 1 year ago

Thanks for the explanations but I still don't fully understand the situation.

It sounds like this is issue essentially about using remoteServerListenOnSocket while also using useLocalServer so that RemoteCommand will work. That's probably just not going to work, due to the constraints on how things work that come from ssh. In that case, I wonder whether dropping remoteServerListenOnSocket is a solution for you. If there's a concern about security in having the server listen to a port, we have protections around access to the server's port that prevent other users from connecting to it.

Besides that, it sounds like what @ekungurov is saying is that using RemoteCommand for the slurm setup is not sufficient, and I think #1722 essentially covers that issue. I don't really understand it though-

can use a customized RemoteCommand to run 'Code' on a computational node (instead of the node we established ssh connection to) and also to set up our own transparent tunneling between the computational node and the slurm control node, so that the 'Code' running on a computational node could be contacted via socket on a control node. However, we don't have a way for the Visual Studio Code (which is being run on a local PC) to use that particular socket instead of the one it thinks it have to use.

With the RemoteCommand, wouldn't that mean that Remote-SSH connects and runs its setup script in the context of the compute node, and forwards our server's port/socket from the compute node to the local machine? Why do you say you have to set up your own forwarding from the login node?

ekungurov commented 1 year ago

@roblourens on HPC users usually don't ssh directly to a compute node to launch a task. Instead, they ssh to a control node (login node) and then submit tasks to the cluster. Cluster mgmt software automatically choose a compute node (worker node) from the pool of compute nodes which currently have requested resources available.

That's why in #1722 a solution from @samfux84 (edited: this one https://github.com/microsoft/vscode-remote-release/issues/1722#issuecomment-837902373) use ssh -L for tunneling, and solution from @simonbyrne (this one https://github.com/microsoft/vscode-remote-release/issues/1722#issuecomment-1184705403) uses nc I believe for the same purpose.

roblourens commented 1 year ago

on HPC users usually don't ssh directly to a compute node to launch a task. Instead, they ssh to a control node (login node) and then submit tasks to the cluster. Cluster mgmt software automatically choose a compute node (worker node) from the pool of compute nodes which currently have requested resources available.

I understand that. But I guess that even if our setup script runs in the compute node context, ssh still connected to the login node and so the forwarding that ssh sets up only goes as far as the login node, that makes sense.

jasperhyp commented 1 year ago

It sounds like this is issue essentially about using remoteServerListenOnSocket while also using useLocalServer so that RemoteCommand will work. That's probably just not going to work, due to the constraints on how things work that come from ssh. In that case, I wonder whether dropping remoteServerListenOnSocket is a solution for you. If there's a concern about security in having the server listen to a port, we have protections around access to the server's port that prevent other users from connecting to it.

@roblourens Please see my discussion with @tanhakabir at the beginning of this post. Specifically this one. I tried combinations of true/false for those settings but there seems not one combination that would allow RemoteCommand to work.

ekungurov commented 1 year ago

@roblourens

if our setup script runs in the compute node context, ssh still connected to the login node

Yes, correct.

so the forwarding that ssh sets up only goes as far as the login node

samfux84 solution is almost completely semi-manual/manual, and it establish ssh tunnel between "localhost" and the compute node (where https://github.com/coder/code-server aka "VS Code in the browser" is being run). It is supposed that end user will download / copy the generated bash script, and run it on his own local Linux/MacOS/WSL workstation, so the "localhost" is a workstation in this case.

simonbyrne solution, on the other hand, runs yet another copy of sshd as a task (as a user process) on a compute node, and then 'nc' the traffic to that instance of sshd.

My question... I mean, I probably have a question... Is it possible to apply custom RemoteCommand to login node, but once it applied - to connect to a compute node : either directly to the compute node and probably manually (directly - without using ssh/nc tunneling and without entering XXXX-XXXX pin code on https://github.com/login/device thus utilizing some MS Azure tunneling technology), OR if that's not possible, maybe via ssh/nc tunneling but that tunneling being set up automatically (via custom RemoteCommand or other means).

roblourens commented 1 year ago

Please see my discussion with @tanhakabir at the beginning of this post. Specifically https://github.com/microsoft/vscode-remote-release/issues/6086#issuecomment-1007875147. I tried combinations of true/false for those settings but there seems not one combination that would allow RemoteCommand to work.

Thanks- that configuration should work, since those are the defaults for those settings (on mac/linux, and should generally work on windows). If you are still seeing that, I can help troubleshoot if you share the full log from the Remote-SSH output channel. However one possible explanation for that error message is that TCP port forwarding is disabled on your ssh server, if that's the case, you might need to talk to your system administrator, or, you might be stuck with a socket.

My question...

Great question! I really don't know 😁. I suspect that the answer is no, and that the only solution is for a complex custom workaround of the sort that has been mentioned already.

jasperhyp commented 1 year ago

that configuration should work, since those are the defaults for those settings (on mac/linux, and should generally work on windows)

Thanks, could you specify which settings?

roblourens commented 1 year ago

"remote.SSH.remoteServerListenOnSocket": false and "remote.SSH.useLocalServer": true

samfux84 commented 1 year ago

samfux84 solution is almost completely semi-manual/manual, and it establish ssh tunnel between "localhost" and the compute node (where https://github.com/coder/code-server aka "VS Code in the browser" is being run). It is supposed that end user will download / copy the generated bash script, and run it on his own local Linux/MacOS/WSL workstation, so the "localhost" is a workstation in this case.

Hi @ekungurov,

I scripted our solution that we are using for the users of our HPC cluster, so for the end user it is fully automated (including the setup of the SSH tunnel and opening the local browser with the URL from the code-server):

https://gitlab.ethz.ch/sfux/VSCode_remote_HPC

The only prerequisite for the end user is to have SSH keys for password-less access to the cluster (as all commands are run through ssh, because the script runs on the local computer of the user).

The users can download our script and run it on their local computer specifying what resources they want for the batch job and then wait until it has started.

For instance, if I run the command

./start_vscode.sh -u sfux -b SLURM -n 4 -W 04:00 -m 2048

on my local computer, then it will login to the cluster as user sfux, submit a batch job via Slurm, requesting 4 cores, 4 hours of runtime and 2 GB of memory per core, setup the required SSH tunnel and start the local browser with the URL provided by the code-server. We tested the script with Linux, macOS and Windows (using git-bash) and it works.

We plan to replace code-server in the future with vscode-server once there is a public release.

This kind of setup can also be used for other use-cases like jupyter scripts (in case there is no jupyter hub available):

https://gitlab.ethz.ch/sfux/Jupyter-on-Euler-or-Leonhard-Open

jasperhyp commented 1 year ago

"remote.SSH.remoteServerListenOnSocket": false and "remote.SSH.useLocalServer": true

It seems not working though. I can connect to the login node but no compute node is requested, meaning that the command is not run. My setting.json looks like:

"remote.SSH.enableRemoteCommand": true,
"terminal.integrated.inheritEnv": false,
"cmake.configureOnOpen": false,
"remote.SSH.remoteServerListenOnSocket": false,  // changed
"remote.SSH.useLocalServer": true,

I used a very simple setup in my config:

Host some_name
    HostName some_host
    User some_user
    RemoteCommand some_srun_command

Silly question: How to get logs when logging with remote-ssh? It looks like it's now not creating logs by default if the connection is successful.

roblourens commented 1 year ago

View > Output > Remote-SSH in the dropdown

I'd rather discuss the general "how to connect to slurm" issues in #1722 since it seems that's what that issue is generally about. This one can be "using remoteListenOnSocket with useLocalServer", but I'm not sure we'll be able to do anything about that.

jasperhyp commented 1 year ago

I also noticed the error "Cannot use RemoteCommand if you have this remote in your "remote.SSH.remotePlatform" setting", so I followed the solution here to change the RequestTTY and IdentityFile, comment out remote.SSH.remotePlatform, and set remoteListenOnSocket and useLocalServer as you recommended. It then looked like RemoteCommand worked. However, I was not able to connect to the server if using RemoteCommand. It was only when I commented that line in config that I could connect to the server.

Config:

Host test
    HostName some_server
    User yeh803
    IdentityFile C:/Users/xxx/.ssh/id_rsa
    RequestTTY yes  # I also tried commenting this but no difference
    RemoteCommand ls -l

Error log below:

Log ``` [17:04:55.434] Log Level: 2 [17:04:55.441] remote-ssh@0.95.2022121315 [17:04:55.441] win32 x64 [17:04:55.447] SSH Resolver called for "ssh-remote+test", attempt 1 [17:04:55.447] "remote.SSH.useLocalServer": true [17:04:55.448] "remote.SSH.path": undefined [17:04:55.448] "remote.SSH.configFile": undefined [17:04:55.448] "remote.SSH.useFlock": true [17:04:55.448] "remote.SSH.lockfilesInTmp": false [17:04:55.448] "remote.SSH.localServerDownload": auto [17:04:55.448] "remote.SSH.remoteServerListenOnSocket": false [17:04:55.449] "remote.SSH.showLoginTerminal": false [17:04:55.449] "remote.SSH.defaultExtensions": [] [17:04:55.449] "remote.SSH.loglevel": 2 [17:04:55.449] "remote.SSH.enableDynamicForwarding": true [17:04:55.449] "remote.SSH.enableRemoteCommand": true [17:04:55.449] "remote.SSH.serverPickPortsFromRange": {} [17:04:55.449] "remote.SSH.serverInstallPath": {} [17:04:55.455] SSH Resolver called for host: test [17:04:55.455] Setting up SSH remote "test" [17:04:55.460] Acquiring local install lock: C:\Users\xxx\AppData\Local\Temp\vscode-remote-ssh-1d33c4c4-install.lock [17:04:55.464] Looking for existing server data file at c:\Users\xxx\AppData\Roaming\Code\User\globalStorage\ms-vscode-remote.remote-ssh\vscode-ssh-host-1d33c4c4-5235c6bb189b60b01b1f49062f4ffa42384f8c91-0.95.2022121315\data.json [17:04:55.465] Using commit id "5235c6bb189b60b01b1f49062f4ffa42384f8c91" and quality "stable" for server [17:04:55.468] Install and start server if needed [17:04:55.472] Checking ssh with "ssh -V" [17:04:55.505] > OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 [17:04:55.507] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V" [17:04:55.535] > OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 [17:04:55.537] Checking ssh with "C:\Program Files\Git\usr\bin\ssh.exe -V" [17:04:55.538] Got error from ssh: spawn C:\Program Files\Git\usr\bin\ssh.exe ENOENT [17:04:55.539] Checking ssh with "C:\Program Files (x86)\Git\usr\bin\ssh.exe -V" [17:04:55.540] Got error from ssh: spawn C:\Program Files (x86)\Git\usr\bin\ssh.exe ENOENT [17:04:55.540] Checking ssh with "ssh -V" [17:04:55.572] > OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 [17:04:55.577] Using SSH config file "C:\Users\xxx\.ssh\config" [17:04:55.578] askpass server listening on \\.\pipe\vscode-ssh-askpass-f42e2d0385a12d04c58557a66c48cf19c16b34cf-sock [17:04:55.579] Spawning local server with {"serverId":1,"ipcHandlePath":"\\\\.\\pipe\\vscode-ssh-askpass-e5b70cf896caf65d3c6f39c15eaa5baae1d0a91a-sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","12927","-F","C:\\Users\\xxx\\.ssh\\config","test"],"serverDataFolderName":".vscode-server","dataFilePath":"c:\\Users\\xxx\\AppData\\Roaming\\Code\\User\\globalStorage\\ms-vscode-remote.remote-ssh\\vscode-ssh-host-1d33c4c4-5235c6bb189b60b01b1f49062f4ffa42384f8c91-0.95.2022121315\\data.json"} [17:04:55.579] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"c:\\Users\\xxx\\.vscode\\extensions\\ms-vscode-remote.remote-ssh-0.95.2022121315\\out\\local-server\\askpass.bat","VSCODE_SSH_ASKPASS_NODE":"D:\\Program_Files\\Microsoft VS Code\\Code.exe","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"c:\\Users\\xxx\\.vscode\\extensions\\ms-vscode-remote.remote-ssh-0.95.2022121315\\out\\askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"\\\\.\\pipe\\vscode-ssh-askpass-f42e2d0385a12d04c58557a66c48cf19c16b34cf-sock"} [17:04:55.610] Spawned 28132 [17:04:55.812] > local-server-1> Running ssh connection command: "-v -T -D 12927 -F C:\\Users\\xxx\\.ssh\\config test" [17:04:55.820] > local-server-1> Spawned ssh, pid=29516 [17:04:55.885] stderr> OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 [17:04:55.935] stderr> debug1: Server host key: some_host_key [17:04:55.995] stderr> Problems logging in? [17:04:55.995] stderr> Use your lower case ID, like abc123, not ABC123. [17:04:56.495] stderr> Authenticated to some_server (some_ip). [17:04:59.735] > total 64 > drwxr-xr-x 3 yeh803 yeh803 22 Jul 23 22:32 ondemand > drwxrwxr-x 12 yeh803 yeh803 311 Dec 8 07:08 workspace [17:04:59.736] stderr> Transferred: sent 3144, received 3172 bytes, in 3.2 seconds [17:04:59.736] stderr> Bytes per second: sent 971.1, received 979.7 [17:04:59.739] > local-server-1> ssh child died, shutting down [17:04:59.746] Local server exit: 0 [17:04:59.747] Received install output: local-server-1> Running ssh connection command: "-v -T -D 12927 -F C:\\Users\\xxx\\.ssh\\config test" local-server-1> Spawned ssh, pid=29516 OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 debug1: Server host key: some_host_key Problems logging in? Use your lower case ID, like abc123, not ABC123. Authenticated to some_server (some_ip). total 64 drwxr-xr-x 3 yeh803 yeh803 22 Jul 23 22:32 ondemand drwxrwxr-x 12 yeh803 yeh803 311 Dec 8 07:08 workspace Transferred: sent 3144, received 3172 bytes, in 3.2 seconds Bytes per second: sent 971.1, received 979.7 local-server-1> ssh child died, shutting down [17:04:59.747] You have RemoteCommand in your SSH Config for this remote which could be the cause for this crash. Try reconnecting with `enableRemoteCommand` set to `false`. [17:04:59.747] Failed to parse remote port from server output [17:04:59.750] Resolver error: Error: at g.Create (c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:583926) at t.handleInstallOutput (c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:582540) at Object.e [as tryInstallWithLocalServer] (c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:623191) at process.processTicksAndRejections (node:internal/process/task_queues:96:5) at async c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:642446 at async t.withShowDetailsEvent (c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:646166) at async c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:621663 at async T (c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:618169) at async t.resolveWithLocalServer (c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:621278) at async t.resolve (c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:643774) at async c:\Users\xxx\.vscode\extensions\ms-vscode-remote.remote-ssh-0.95.2022121315\out\extension.js:1:723042 [17:04:59.754] ##------ ```
roblourens commented 1 year ago

RemoteCommand has to be something that won't terminate, like ls -l; bash

jasperhyp commented 1 year ago

Doesn't work, same error log.

gvegayon commented 1 year ago

As of June 21, creating a direct connection to a Slurm job is still faulty.

Version: 1.80.0-insider
Commit: 79c4092768e41b16fd823e80c874dbfafdc08f55
Date: 2023-06-13T05:22:36.374Z
Electron: 22.5.5
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: 10.8.168.25-electron.0
OS: Linux x64 5.19.0-45-generic

This would be AWESOME.

moon6pence commented 4 months ago

Followed up all comments in 2024! And agree for your point @roblourens

This one can be "using remoteListenOnSocket with useLocalServer", but I'm not sure we'll be able to do anything about that.

I think main issue we discuss here is that,

Q1. Do you think above setting is completely unavailable by design? Q2. Is there yet another solution to let vscode communicate properly when RemoteCommand actually start bash in other host?

IMHO if there is no good way for both question we'd better close this ticket and keep discussing in main issue for HPC usage demands https://github.com/microsoft/vscode-remote-release/issues/1722