microsoft / vscode-remote-release

Visual Studio Code Remote Development: Open any folder in WSL, in a Docker container, or on a remote machine using SSH and take advantage of VS Code's full feature set.
https://aka.ms/vscode-remote
Other
3.6k stars 273 forks source link

[Remote-SSH Bug]: Remote SSH fails at "Waiting for server log..." #8063

Closed keyonvafa closed 1 year ago

keyonvafa commented 1 year ago

Is there an existing issue for this bug?

Required Troubleshooting Steps

Connect Locally

It doesn’t connect successfully (same error message)

->

No response

Expected Behavior

I've been using Remote for the past two years to SSH in to another server without problems. However in the last week I am suddenly unable to connect. I've tried the usual suspects to no avail:

Moreover, this issue doesn't occur when I ssh to the server via the command line -- it only happens when I connect with Remote.

Steps To Reproduce

When I try connecting to Remote, the connection hangs -- I enter my password and provide 2 factor authentication successfully, but it gets stuck at "Waiting for server log..."

Remote-SSH Log

Remote-SSH Log

``` [22:25:31.454] Log Level: 2 [22:25:31.462] SSH Resolver called for "ssh-remote+sh03-ln06.stanford.edu", attempt 1 [22:25:31.462] "remote.SSH.useLocalServer": true [22:25:31.463] "remote.SSH.path": undefined [22:25:31.463] "remote.SSH.configFile": undefined [22:25:31.463] "remote.SSH.useFlock": true [22:25:31.463] "remote.SSH.lockfilesInTmp": false [22:25:31.463] "remote.SSH.localServerDownload": auto [22:25:31.463] "remote.SSH.remoteServerListenOnSocket": false [22:25:31.463] "remote.SSH.showLoginTerminal": false [22:25:31.463] "remote.SSH.defaultExtensions": [] [22:25:31.464] "remote.SSH.loglevel": 2 [22:25:31.464] "remote.SSH.enableDynamicForwarding": true [22:25:31.464] "remote.SSH.enableRemoteCommand": false [22:25:31.464] "remote.SSH.serverPickPortsFromRange": {} [22:25:31.464] "remote.SSH.serverInstallPath": {} [22:25:31.466] VS Code version: 1.75.1 [22:25:31.466] Remote-SSH version: remote-ssh@0.96.0 [22:25:31.466] darwin x64 [22:25:31.469] SSH Resolver called for host: sh03-ln06.stanford.edu [22:25:31.469] Setting up SSH remote "sh03-ln06.stanford.edu" [22:25:31.472] Acquiring local install lock: /var/folders/l8/kddsf89n7vz079xfph1jpwq40000gn/T/vscode-remote-ssh-106faef7-install.lock [22:25:31.472] Looking for existing server data file at /Users/keyonvafa/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-106faef7-441438abd1ac652551dbe4d408dfcec8a499b8bf-0.96.0/data.json [22:25:31.473] Using commit id "441438abd1ac652551dbe4d408dfcec8a499b8bf" and quality "stable" for server [22:25:31.475] Install and start server if needed [22:25:31.481] PATH: /usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.7.0/bin:/Users/keyonvafa/.gem/ruby/2.6.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin [22:25:31.481] Checking ssh with "ssh -V" [22:25:31.490] > OpenSSH_8.4p1, OpenSSL 1.1.1l 24 Aug 2021 [22:25:31.496] askpass server listening on /var/folders/l8/kddsf89n7vz079xfph1jpwq40000gn/T/vscode-ssh-askpass-d6994ed54907c097c4d863d03857b0f3b66ed7e2.sock [22:25:31.496] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/l8/kddsf89n7vz079xfph1jpwq40000gn/T/vscode-ssh-askpass-f37b19268e2f529f3775a67e7fca0d7fb9aaa284.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","57141","-o","ConnectTimeout=30","sh03-ln06.stanford.edu"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/keyonvafa/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-106faef7-441438abd1ac652551dbe4d408dfcec8a499b8bf-0.96.0/data.json"} [22:25:31.496] Local server env: {"SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.3aeNsqbVj7/Listeners","SHELL":"/bin/zsh","DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/l8/kddsf89n7vz079xfph1jpwq40000gn/T/vscode-ssh-askpass-d6994ed54907c097c4d863d03857b0f3b66ed7e2.sock"} [22:25:31.497] Spawned 10658 [22:25:31.654] > local-server-1> Running ssh connection command: "-v -T -D 57141 -o ConnectTimeout=30 sh03-ln06.stanford.edu" [22:25:31.659] > local-server-1> Spawned ssh, pid=10667 [22:25:31.667] stderr> OpenSSH_8.4p1, OpenSSL 1.1.1l 24 Aug 2021 [22:25:31.988] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:eB0bODKdaCWtPgv0pYozsdC5ckfcBFVOxeMwrNKdkmg [22:25:32.506] stderr> Authenticated with partial success. [22:25:32.828] Got askpass request: {"request":"Duo two-factor login for keyon Enter a passcode or select one of the following options: 1. Duo Push to XXX-XXX-XXXX 2. Duo Push to iOS 3. Phone call to XXX-XXX-XXXX 4. SMS passcodes to XXX-XXX-XXXX Passcode or option (1-4):"} [22:25:32.829] Listening for interwindow password on /var/folders/l8/kddsf89n7vz079xfph1jpwq40000gn/T/vscode-ssh-askpass-47d49a84088ebd84f8456d594b86114e576a66f1.sock [22:25:32.829] Writing password prompt to globalState [22:25:33.525] Interactor gave response: * [22:25:33.526] Cleaning up other-window auth server [22:25:38.362] stderr> Authenticated to sh03-ln06.stanford.edu ([171.66.103.164]:22). [22:25:38.363] stderr> bind [::1]:16001: Address already in use [22:25:38.364] stderr> bind [127.0.0.1]:16001: Address already in use [22:25:38.364] stderr> channel_setup_fwd_listener_tcpip: cannot listen to port: 16001 [22:25:38.364] stderr> bind [::1]:16002: Address already in use [22:25:38.364] stderr> bind [127.0.0.1]:16002: Address already in use [22:25:38.364] stderr> channel_setup_fwd_listener_tcpip: cannot listen to port: 16002 [22:25:38.365] stderr> bind [::1]:16003: Address already in use [22:25:38.365] stderr> bind [127.0.0.1]:16003: Address already in use [22:25:38.365] stderr> channel_setup_fwd_listener_tcpip: cannot listen to port: 16003 [22:25:38.365] stderr> bind [::1]:16004: Address already in use [22:25:38.366] stderr> bind [127.0.0.1]:16004: Address already in use [22:25:38.366] stderr> channel_setup_fwd_listener_tcpip: cannot listen to port: 16004 [22:25:38.366] stderr> bind [::1]:16005: Address already in use [22:25:38.366] stderr> bind [127.0.0.1]:16005: Address already in use [22:25:38.366] stderr> channel_setup_fwd_listener_tcpip: cannot listen to port: 16005 [22:25:38.586] > Success. Logging you in... > Last failed login: Sun Feb 19 19:22:14 PST 2023 from pool-173-52-222-58.nycmny.fios.verizon.net on ssh:notty > There was 1 failed login attempt since the last successful login. [22:25:39.153] stderr> The following modules were not unloaded: [22:25:39.153] stderr> (Use "module --force purge" to unload all): [22:25:39.153] stderr> [22:25:39.154] stderr> 1) devel 2) math [22:25:39.517] > ready: e5ab059fc972 [22:25:39.594] > Linux 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 [22:25:39.594] Platform: linux [22:25:39.669] > /bin/bash [22:25:39.670] Shell: bash [22:25:39.670] Creating bash subshell inside "bash" [22:25:39.753] > bash version: 4.2.46(2)-release [22:25:39.797] > bash version: 4.2.46(2)-release [22:25:39.835] > e5ab059fc972: running [22:25:39.874] > Acquiring lock on /home/users/keyon/.vscode-server/bin/441438abd1ac652551dbe4d408dfcec8a499b8bf/vscode-remote-lock.keyon.441438abd1ac652551dbe4d408dfcec8a499b8bf [22:25:39.879] > Installing to /home/users/keyon/.vscode-server/bin/441438abd1ac652551dbe4d408dfcec8a499b8bf... [22:25:39.882] > e5ab059fc972%%1%% [22:25:39.889] > Downloading with wget [22:25:40.739] > Download complete [22:25:40.745] > e5ab059fc972%%2%% > tar --version: [22:25:40.746] > tar (GNU tar) 1.26 > Copyright (C) 2011 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later . > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > > Written by John Gilmore and Jay Fenlason. [22:25:53.035] > Checking /home/users/keyon/.vscode-server/.441438abd1ac652551dbe4d408dfcec8a499b8bf.log and /home/users/keyon/.vscode-server/.441438abd1ac652551dbe4d408dfcec8a499b8bf.pid for a running server [22:25:53.036] > Running ssh connection command... /home/users/keyon/.vscode-server/bin/441438abd1ac652551dbe4d408dfcec8a499b8bf/bin/code-server --start-server --host=127.0.0.1 --accept-server-license-terms --enable-remote-auto-shutdown --port=0 --telemetry-level all &> "/home/users/keyon/.vscode-server/.441438abd1ac652551dbe4d408dfcec8a499b8bf.log" < /dev/null > printenv: > LMOD_SYSHOST=sherlock > MANPATH=/share/software/user/open/python/3.9.0/share/man:/share/software/user/open/libffi/3.2.1/share/man:/share/software/user/open/sqlite/3.37.2/share/man:/share/software/user/open/readline/7.0/share/man:/share/software/user/open/tcltk/8.6.6/share/man:/share/software/user/open/libressl/3.2.1/share/man:/share/software/user/open/zlib/1.2.11/share/man:/share/software/user/open/lmod/lmod/share/man:/usr/local/share/man:/usr/share/man:/opt/dell/srvadmin/share/man > HOSTNAME=sh03-ln06.stanford.edu > _ModuleTable003_=ck5hbWUgPSAibGlicmVzc2wvMy4yLjEiLAp3ViA9ICIwMDAwMDAwMDMuMDAwMDAwMDAyLjAwMDAwMDAwMS4qemZpbmFsIiwKfSwKbWF0aCA9IHsKZm4gPSAiL3NoYXJlL3NvZnR3YXJlL21vZHVsZXMvY2F0ZWdvcmllcy9tYXRoLmx1YSIsCmZ1bGxOYW1lID0gIm1hdGgiLApsb2FkT3JkZXIgPSAyLApwcm9wVCA9IHsKbG1vZCA9IHsKc3RpY2t5ID0gMSwKfSwKfSwKc3RhY2tEZXB0aCA9IDAsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJtYXRoIiwKd1YgPSAiTS4qemZpbmFsIiwKfSwKcHl0aG9uID0gewpmbiA9ICIvc2hhcmUvc29mdHdhcmUvbW9kdWxlcy9kZXZlbC9weXRob24vMy45LjAubHVhIiwKZnVsbE5hbWUgPSAicHl0aG9uLzMuOS4wIiwKbG9hZE9yZGVy > SHELL=/bin/bash > __LMOD_REF_COUNT_MODULEPATH=/share/software/modules/system:1;/share/software/modules/math:1;/share/software/modules/devel:1;/share/software/modules/categories:1 > HISTSIZE=1000 > LMOD_ROOT=/share/software/user/open/lmod > APPTAINER_TMPDIR=/tmp > SSH_CLIENT=173.52.222.58 57142 22 > MODULEPATH_ROOT=/share/software/modules > LMOD_SYSTEM_DEFAULT_MODULES=devel,math > TMPDIR=/tmp > APPTAINER_UNSHARE_IPC=true > LMOD_PACKAGE_PATH=/share/software/modules/ > LIBRARY_PATH=/share/software/user/open/python/3.9.0/lib:/share/software/user/open/libffi/3.2.1/lib64:/share/software/user/open/sqlite/3.37.2/lib:/share/software/user/open/readline/7.0/lib:/share/software/user/open/tcltk/8.6.6/lib:/share/software/user/open/libressl/3.2.1/lib:/share/software/user/open/zlib/1.2.11/lib > LMOD_PKG=/share/software/user/open/lmod/lmod > L_SCRATCH=/lscratch/keyon > SHERLOCK=2 > LMOD_VERSION=8.7.14 > LMOD_ADMIN_FILE=/share/software/modules/admin.list > __LMOD_REF_COUNT_CMAKE_PREFIX_PATH=/share/software/user/open/tcltk/8.6.6:1 > _ModuleTable007_=cHJvcFQgPSB7fSwKcmVmX2NvdW50ID0gMSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJ6bGliLzEuMi4xMSIsCndWID0gIjAwMDAwMDAwMS4wMDAwMDAwMDIuMDAwMDAwMDExLip6ZmluYWwiLAp9LAp9LAptcGF0aEEgPSB7CiIvc2hhcmUvc29mdHdhcmUvbW9kdWxlcy9zeXN0ZW0iLCAiL3NoYXJlL3NvZnR3YXJlL21vZHVsZXMvbWF0aCIKLCAiL3NoYXJlL3NvZnR3YXJlL21vZHVsZXMvZGV2ZWwiLCAiL3NoYXJlL3NvZnR3YXJlL21vZHVsZXMvY2F0ZWdvcmllcyIsCn0sCnN5c3RlbUJhc2VNUEFUSCA9ICIvc2hhcmUvc29mdHdhcmUvbW9kdWxlcy9jYXRlZ29yaWVzIiwKfQo= > USER=keyon > GROUP=athey > LS_COLORS=su=00:sg=00:ca=00:or=40;31;01 > LMOD_sys=Linux > LD_LIBRARY_PATH=/share/software/user/open/python/3.9.0/lib:/share/software/user/open/libffi/3.2.1/lib64:/share/software/user/open/sqlite/3.37.2/lib:/share/software/user/open/readline/7.0/lib:/share/software/user/open/tcltk/8.6.6/lib:/share/software/user/open/libressl/3.2.1/lib:/share/software/user/open/zlib/1.2.11/lib > LMOD_MODULERCFILE=/share/software/modules/.modulerc.lua > CPATH=/share/software/user/open/python/3.9.0/include/python3.9:/share/software/user/open/python/3.9.0/include:/share/software/user/open/libffi/3.2.1/lib/libffi-3.2.1/include:/share/software/user/open/sqlite/3.37.2/include:/share/software/user/open/readline/7.0/include:/share/software/user/open/tcltk/8.6.6/include:/share/software/user/open/libressl/3.2.1/include:/share/software/user/open/zlib/1.2.11/include > _ModuleTable004_=ID0gMTAsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAicHl0aG9uLzMuOS4wIiwKd1YgPSAiMDAwMDAwMDAzLjAwMDAwMDAwOS4qemZpbmFsIiwKfSwKcmVhZGxpbmUgPSB7CmZuID0gIi9zaGFyZS9zb2Z0d2FyZS9tb2R1bGVzL3N5c3RlbS9yZWFkbGluZS83LjAubHVhIiwKZnVsbE5hbWUgPSAicmVhZGxpbmUvNy4wIiwKbG9hZE9yZGVyID0gNywKcHJvcFQgPSB7fSwKcmVmX2NvdW50ID0gMSwKc3RhY2tEZXB0aCA9IDIsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJyZWFkbGluZS83LjAiLAp3ViA9ICIwMDAwMDAwMDcuKnpmaW5hbCIsCn0sCnNxbGl0ZSA9IHsKZm4gPSAiL3NoYXJlL3NvZnR3YXJlL21v > TMOUT=86400 > GROUP_SCRATCH=/scratch/groups/a [22:25:53.110] > they > VSCODE_AGENT_FOLDER=/home/users/keyon/.vscode-server > LMOD_PREPEND_BLOCK=normal > APPTAINER_CACHEDIR=/scratch/users/keyon/.apptainer > PATH=/share/software/user/open/python/3.9.0/bin:/share/software/user/open/sqlite/3.37.2/bin:/share/software/user/open/tcltk/8.6.6/bin:/share/software/user/open/libressl/3.2.1/bin:/share/software/user/srcc/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/iSM/bin:/home/users/keyon/.local/bin:/home/users/keyon/bin > MAIL=/var/spool/mail/keyon > SRCC_PATH=/share/software/user/srcc/bin > _ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpweXRob24gPSAicHl0aG9uIiwKfSwKbVQgPSB7CmRldmVsID0gewpmbiA9ICIvc2hhcmUvc29mdHdhcmUvbW9kdWxlcy9jYXRlZ29yaWVzL2RldmVsLmx1YSIsCmZ1bGxOYW1lID0gImRldmVsIiwKbG9hZE9yZGVyID0gMSwKcHJvcFQgPSB7Cmxtb2QgPSB7CnN0aWNreSA9IDEsCn0sCn0sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAiZGV2ZWwiLAp3ViA9ICJNLip6ZmluYWwiLAp9LApsaWJmZmkgPSB7CmZuID0gIi9zaGFyZS9zb2Z0d2FyZS9tb2R1bGVzL3N5 > sherlock_FAMILY_PYTHON=python > OAK=/oak/stanford/groups/athey > LMOD_SETTARG_CMD=: > PWD=/home/users/keyon > GROUP_HOME=/home/groups/athey > _LMFILES_=/share/software/modules/categories/devel.lua:/share/software/modules/categories/math.lua:/share/software/modules/categories/system.lua:/share/software/modules/system/zlib/1.2.11.lua:/share/software/modules/system/libressl/3.2.1.lua:/share/software/modules/system/tcltk/8.6.6.lua:/share/software/modules/system/readline/7.0.lua:/share/software/modules/system/sqlite/3.37.2.lua:/share/software/modules/system/libffi/3.2.1.lua:/share/software/modules/devel/python/3.9.0.lua > LOCAL_SCRATCH=/lscratch/keyon > LANG=en_US.UTF-8 > MODULEPATH=/share/software/modules/system:/share/software/modules/math:/share/software/modules/devel:/share/software/modules/categories > LOADEDMODULES=devel:math:system:zlib/1.2.11:libressl/3.2.1:tcltk/8.6.6:readline/7.0:sqlite/3.37.2:libffi/3.2.1:python/3.9.0 > _ModuleTable_Sz_=7 > LMOD_CMD=/share/software/user/open/lmod/lmod/libexec/lmod > LMOD_FAMILY_PYTHON_VERSION=3.9.0 > _ModuleTable005_=ZHVsZXMvc3lzdGVtL3NxbGl0ZS8zLjM3LjIubHVhIiwKZnVsbE5hbWUgPSAic3FsaXRlLzMuMzcuMiIsCmxvYWRPcmRlciA9IDgsCnByb3BUID0ge30sCnJlZl9jb3VudCA9IDEsCnN0YWNrRGVwdGggPSAxLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAic3FsaXRlLzMuMzcuMiIsCndWID0gIjAwMDAwMDAwMy4wMDAwMDAwMzcuMDAwMDAwMDAyLip6ZmluYWwiLAp9LApzeXN0ZW0gPSB7CmZuID0gIi9zaGFyZS9zb2Z0d2FyZS9tb2R1bGVzL2NhdGVnb3JpZXMvc3lzdGVtLmx1YSIsCmZ1bGxOYW1lID0gInN5c3RlbSIsCmxvYWRPcmRlciA9IDMsCnByb3BUID0ge30sCnJlZl9jb3VudCA9IDEsCnN0YWNrRGVwdGggPSAxLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5h > LMOD_AVAIL_STYLE=categories > KRB5CCNAME=FILE:/tmp/krb5cc_392801_BhqHU1vG3m > HISTCONTROL=ignoreboth:erasedups > LMOD_REDIRECT=yes > LMOD_FAMILY_PYTHON=python > HOME=/home/users/keyon > SHLVL=2 > __LMOD_REF_COUNT_PATH=/share/software/user/open/python/3.9.0/bin:1;/share/software/user/open/sqlite/3.37.2/bin:1;/share/software/user/open/tcltk/8.6.6/bin:1;/share/software/user/open/libressl/3.2.1/bin:1;/share/software/user/srcc/bin:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/opt/dell/srvadmin/bin:1;/opt/dell/srvadmin/iSM/bin:1 > __LMOD_REF_COUNT_CPATH=/share/software/user/open/python/3.9.0/include/python3.9:1;/share/software/user/open/python/3.9.0/include:1;/share/software/user/open/libffi/3.2.1/lib/libffi-3.2.1/include:1;/share/software/user/open/sqlite/3.37.2/include:1;/share/software/user/open/readline/7.0/include:1;/share/software/user/open/tcltk/8.6.6/include:1;/share/software/user/open/libressl/3.2.1/include:1;/share/software/user/open/zlib/1.2.11/include:1 > _ModuleTable002_=c3RlbS9saWJmZmkvMy4yLjEubHVhIiwKZnVsbE5hbWUgPSAibGliZmZpLzMuMi4xIiwKbG9hZE9yZGVyID0gOSwKcHJvcFQgPSB7fSwKcmVmX2NvdW50ID0gMSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJsaWJmZmkvMy4yLjEiLAp3ViA9ICIwMDAwMDAwMDMuMDAwMDAwMDAyLjAwMDAwMDAwMS4qemZpbmFsIiwKfSwKbGlicmVzc2wgPSB7CmZuID0gIi9zaGFyZS9zb2Z0d2FyZS9tb2R1bGVzL3N5c3RlbS9saWJyZXNzbC8zLjIuMS5sdWEiLApmdWxsTmFtZSA9ICJsaWJyZXNzbC8zLjIuMSIsCmxvYWRPcmRlciA9IDUsCnByb3BUID0ge30sCnJlZl9jb3VudCA9IDEsCnN0YWNrRGVwdGggPSAxLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNl > BASH_ENV=/share/software/user/open/lmod/lmod/init/bash > LOGNAME=keyon > LMOD_arch=x86_64 > LMOD_SHELL_PRGM=bash > SSH_CONNECTION=173.52.222.58 57142 171.66.103.164 22 > MODULESHOME=/share/software/user/open/lmod/lmod > LMOD_RC=/share/software/modules/lmodrc.lua > __LMOD_REF_COUNT_LIBRARY_PATH=/share/software/user/open/python/3.9.0/lib:1;/share/software/user/open/libffi/3.2.1/lib64:1;/share/software/user/open/sqlite/3.37.2/lib:1;/share/software/user/open/readline/7.0/lib:1;/share/software/user/open/tcltk/8.6.6/lib:1;/share/software/user/open/libressl/3.2.1/lib:1;/share/software/user/open/zlib/1.2.11/lib:1 > LESSOPEN=||/usr/bin/lesspipe.sh %s > LMOD_SETTARG_FULL_SUPPORT=no > __LMOD_REF_COUNT_LD_LIBRARY_PATH=/share/software/user/open/python/3.9.0/lib:1;/share/software/user/open/libffi/3.2.1/lib64:1;/share/software/user/open/sqlite/3.37.2/lib:1;/share/software/user/open/readline/7.0/lib:1;/share/software/user/open/tcltk/8.6.6/lib:1;/share/software/user/open/libressl/3.2.1/lib:1;/share/software/user/open/zlib/1.2.11/lib:1 > PKG_CONFIG_PATH=/share/software/user/open/python/3.9.0/lib/pkgconfig:/share/software/user/open/libffi/3.2.1/lib/pkgconfig:/share/software/user/open/sqlite/3.37.2/lib/pkgconfig:/share/software/user/open/tcltk/8.6.6/lib/pkgconfig:/share/software/user/open/libressl/3.2.1/lib/pkgconfig > SH_TIMEOUT_CMD=timeout -s9 > PROMPT_COMMAND=RET=$?;/bin/logger -t user_audit "username=$USER pid=$$ cmd=\"$(history 1 | /bin/sed "s/^[ ]*[0-9]\+[ ]*//" )\" newpwd=$PWD ret=$RET" 2>/dev/null > __Init_Default_Modules=1 > LMOD_FULL_SETTARG_SUPPORT=no > XDG_RUNTIME_DIR=/tmp > CMAKE_PREFIX_PATH=/share/software/user/open/tcltk/8.6.6 > __LMOD_REF_COUNT_PKG_CONFIG_PATH=/share/software/user/open/python/3.9.0/lib/pkgconfig:1;/share/software/user/open/libffi/3.2.1/lib/pkgconfig:1;/share/software/user/open/sqlite/3.37.2/lib/pkgconfig:1;/share/software/user/open/tcltk/8.6.6/lib/pkgconfig:1;/share/software/user/open/libressl/3.2.1/lib/pkgconfig:1 > LMOD_DIR=/share/software/user/open/lmod/lmod/libexec > __LMOD_REF_COUNT_MANPATH=/share/software/user/open/python/3.9.0/share/man:1;/share/software/user/open/libffi/3.2.1/share/man:1;/share/software/user/open/sqlite/3.37.2/share/man:1;/share/software/user/open/readline/7.0/share/man:1;/share/software/user/open/tcltk/8.6.6/share/man:1;/share/software/user/open/libressl/3.2.1/share/man:1;/share/software/user/open/zlib/1.2.11/share/man:1;/share/software/user/open/lmod/lmod/share/man:1;/usr/local/share/man:1;/usr/share/man:1;/opt/dell/srvadmin/share/man:1 > _ModuleTable006_=bWUgPSAic3lzdGVtIiwKd1YgPSAiTS4qemZpbmFsIiwKfSwKdGNsdGsgPSB7CmZuID0gIi9zaGFyZS9zb2Z0d2FyZS9tb2R1bGVzL3N5c3RlbS90Y2x0ay84LjYuNi5sdWEiLApmdWxsTmFtZSA9ICJ0Y2x0ay84LjYuNiIsCmxvYWRPcmRlciA9IDYsCnByb3BUID0ge30sCnJlZl9jb3VudCA9IDEsCnN0YWNrRGVwdGggPSAxLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAidGNsdGsvOC42LjYiLAp3ViA9ICIwMDAwMDAwMDguMDAwMDAwMDA2LjAwMDAwMDAwNi4qemZpbmFsIiwKfSwKemxpYiA9IHsKZm4gPSAiL3NoYXJlL3NvZnR3YXJlL21vZHVsZXMvc3lzdGVtL3psaWIvMS4yLjExLmx1YSIsCmZ1bGxOYW1lID0gInpsaWIvMS4yLjExIiwKbG9hZE9yZGVyID0gNCwK > sherlock_FAMILY_PYTHON_VERSION=3.9.0 > SH_DEF_TIMEOUT=30 > LMOD_COLORIZE=yes > SCRATCH=/scratch/users/keyon > BASH_FUNC__cache_cmd()=() { local cache=$HOME/.cache/sh/; > local cmd=${1:-}; > local cachelife=${2:-}; > local entry exp; > shift 2; > [[ -d ${cache} ]] || /bin/mkdir -p "${cache}"; > read -r prm_hash dash <<< "$(/bin/md5sum <<< "$@")"; > entry="${cache}/${cmd##*/}_${prm_hash}"; > exp=$(/bin/date -d -"${cachelife}"sec +%s); > if [[ -s ${entry} ]] && [[ $(/bin/date -r "${entry}" +%s) -gt $exp ]]; then > cat "${entry}"; > else > ${cmd} "$@" 2>&1 | tee "${entry}" 2> /dev/null || rm "${entry}" 2> /dev/null; > fi; > $SH_TIMEOUT_CMD 5 find "${cache}" -type f -name "${cmd##*/}_*" \! -newermt @"$exp" -delete &>/dev/null > } > BASH_FUNC_sh_quota [22:25:53.111] > ()=() { _cache_cmd $SRCC_PATH/sh_quota 180 "$@" > } > BASH_FUNC_sstat()=() { _cache_cmd /usr/bin/sstat 10 "$@" > } > BASH_FUNC_sh_usage()=() { _cache_cmd $SRCC_PATH/sh_usage 90 "$@" > } > BASH_FUNC_sh_jobwait()=() { _cache_cmd $SRCC_PATH/sh_jobwait 300 "$@" > } > BASH_FUNC_squeue()=() { _cache_cmd /usr/bin/squeue 20 "$@" > } > BASH_FUNC_module()=() { if [ -z "${LMOD_SH_DBG_ON+x}" ]; then > case "$-" in > *v*x*) > __lmod_sh_dbg='vx' > ;; > *v*) > __lmod_sh_dbg='v' > ;; > *x*) > __lmod_sh_dbg='x' > ;; > esac; > fi; > if [ -n "${__lmod_sh_dbg:-}" ]; then > set +$__lmod_sh_dbg; > echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output" 1>&2; > fi; > eval "$($LMOD_CMD $LMOD_SHELL_PRGM "$@")" && eval "$(${LMOD_SETTARG_CMD:-:} -s sh)"; > __lmod_my_status=$?; > if [ -n "${__lmod_sh_dbg:-}" ]; then > echo "Shell debugging restarted" 1>&2; > set -$__lmod_sh_dbg; > fi; > unset __lmod_sh_dbg; > return $__lmod_my_status > } > BASH_FUNC_sh_part()=() { _cache_cmd $SRCC_PATH/sh_part 30 "$@" > } > BASH_FUNC_sh_next_downtime()=() { _cache_cmd $SRCC_PATH/sh_next_downtime 3600 "$@" > } > BASH_FUNC_sudo()=() { $SRCC_PATH/sudo "$@" > } > BASH_FUNC_bc()=() { timeout $TMOUT bc "$@" > } > BASH_FUNC_sinfo()=() { _cache_cmd /usr/bin/sinfo 60 "$@" > } > BASH_FUNC_sacct()=() { _cache_cmd /usr/bin/sacct 60 "$@" > } > BASH_FUNC_sh_jobs()=() { _cache_cmd $SRCC_PATH/sh_jobs 90 "$@" > } > BASH_FUNC_sh_status()=() { _cache_cmd $SRCC_PATH/sh_status 3600 "$@" > } > BASH_FUNC_ml()=() { eval "$($LMOD_DIR/ml_cmd "$@")" > } > _=/usr/bin/printenv > OLDPWD=/home/users/keyon/.vscode-server/bin/441438abd1ac652551dbe4d408dfcec8a499b8bf > Spawned remote server: 92477 > Waiting for server log... > Waiting for server log... > Waiting for server log... [22:25:53.112] stderr> rm: cannot remove ‘/home/users/keyon/.vscode-server/.441438abd1ac652551dbe4d408dfcec8a499b8bf.token’: No such file or directory [22:25:53.112] stderr> bash: line 504: 92477 Segmentation fault $VSCH_SERVER_SCRIPT --start-server --host=127.0.0.1 --accept-server-license-terms --enable-remote-auto-shutdown $PORT_FLAG $TELEMETRY $SERVER_LISTEN_FLAG $EXTENSIONS --connection-token-file $VSCH_TOKENFILE &>"$VSCH_LOGFILE" < /dev/null [22:25:53.121] > Waiting for server log... [22:25:53.156] > Waiting for server log... [22:25:53.190] > Waiting for server log... [22:25:53.224] > Waiting for server log... [22:25:53.259] > Waiting for server log... [22:25:53.292] > Waiting for server log... [22:25:53.326] > Waiting for server log... [22:25:53.361] > Waiting for server log... [22:25:53.394] > Waiting for server log... [22:25:53.429] > Waiting for server log... [22:25:53.463] > Waiting for server log... [22:25:53.496] > Waiting for server log... [22:25:53.530] > Waiting for server log... [22:25:53.564] > Waiting for server log... [22:25:53.598] > Waiting for server log... [22:25:53.632] > Waiting for server log... [22:25:53.666] > Waiting for server log... [22:25:53.699] > Waiting for server log... [22:25:53.734] > Waiting for server log... [22:25:53.769] > Waiting for server log... [22:25:53.802] > Waiting for server log... [22:25:53.836] > Waiting for server log... [22:25:53.871] > Waiting for server log... [22:25:53.904] > Waiting for server log... [22:25:53.938] > Waiting for server log... [22:25:53.974] > Waiting for server log... [22:25:54.006] > Waiting for server log... [22:25:54.041] > Waiting for server log... [22:25:54.075] > Waiting for server log... [22:25:54.109] > Waiting for server log... [22:25:54.142] > Waiting for server log... [22:25:54.176] > Waiting for server log... [22:25:54.210] > Waiting for server log... [22:25:54.245] > Waiting for server log... [22:25:54.279] > Waiting for server log... [22:25:54.313] > Waiting for server log... [22:25:54.347] > Waiting for server log... [22:25:54.381] > Waiting for server log... [22:25:54.415] > Waiting for server log... [22:25:54.449] > Waiting for server log... [22:25:54.484] > Waiting for server log... [22:25:54.517] > Waiting for server log... [22:25:54.550] > Waiting for server log... [22:25:54.585] > Waiting for server log... [22:25:54.618] > Waiting for server log... [22:25:54.652] > Waiting for server log... [22:25:54.688] > Waiting for server log... [22:25:54.719] > Waiting for server log... [22:25:54.753] > Waiting for server log... [22:25:54.790] > Waiting for server log... [22:25:54.821] > Waiting for server log... [22:25:54.855] > Waiting for server log... [22:25:54.889] > Waiting for server log... [22:25:54.923] > Waiting for server log... [22:25:54.957] > Waiting for server log... [22:25:54.991] > Waiting for server log... [22:25:55.024] > Waiting for server log... [22:25:55.058] > Waiting for server log... [22:25:55.092] > Waiting for server log... [22:25:55.125] > Waiting for server log... [22:25:55.159] > Waiting for server log... [22:25:55.194] > Waiting for server log... [22:25:55.227] > Waiting for server log... [22:25:55.261] > Waiting for server log... [22:25:55.295] > Waiting for server log... [22:25:55.328] > Waiting for server log... [22:25:55.363] > Waiting for server log... [22:25:55.398] > Waiting for server log... [22:25:55.431] > Waiting for server log... [22:25:55.465] > Waiting for server log... [22:25:55.499] > Waiting for server log... [22:25:55.532] > Waiting for server log... [22:25:55.567] > Waiting for server log... [22:25:55.601] > Waiting for server log... [22:25:55.635] > Waiting for server log... [22:25:55.669] > Waiting for server log... [22:25:55.703] > Waiting for server log... [22:25:55.736] > Waiting for server log... [22:25:55.770] > Waiting for server log... [22:25:55.805] > Waiting for server log... [22:25:55.838] > Waiting for server log... [22:25:55.872] > Waiting for server log... [22:25:55.906] > Waiting for server log... [22:25:55.940] > Waiting for server log... [22:25:55.974] > Waiting for server log... [22:25:56.008] > Waiting for server log... [22:25:56.042] > Waiting for server log... [22:25:56.075] > Waiting for server log... [22:25:56.110] > Waiting for server log... [22:25:56.143] > Waiting for server log... [22:25:56.177] > Waiting for server log... [22:25:56.212] > Waiting for server log... [22:25:56.246] > Waiting for server log... [22:25:56.280] > Waiting for server log... [22:25:56.314] > Waiting for server log... [22:25:56.347] > Waiting for server log... [22:25:56.381] > Waiting for server log... [22:25:56.416] > Waiting for server log... [22:25:56.449] > Waiting for server log... [22:25:56.483] > Waiting for server log... [22:25:56.517] > Waiting for server log... [22:25:56.551] > Waiting for server log... [22:25:56.584] > Waiting for server log... [22:25:56.619] > Waiting for server log... [22:25:56.652] > Waiting for server log... [22:25:56.687] > Waiting for server log... [22:25:56.721] > Waiting for server log... [22:25:56.755] > Waiting for server log... [22:25:56.789] > Waiting for server log... [22:25:56.823] > Waiting for server log... [22:25:56.857] > Waiting for server log... [22:25:56.891] > Waiting for server log... [22:25:56.925] > Waiting for server log... [22:25:56.960] > Waiting for server log... [22:25:56.994] > Waiting for server log... [22:25:57.029] > Waiting for server log... [22:25:57.061] > Waiting for server log... [22:25:57.096] > Waiting for server log... [22:25:57.131] > Waiting for server log... [22:25:57.164] > Waiting for server log... [22:25:57.198] > Waiting for server log... [22:25:57.232] > Waiting for server log... [22:25:57.266] > Waiting for server log... [22:25:57.300] > Waiting for server log... [22:25:57.334] > Waiting for server log... [22:25:57.368] > Waiting for server log... [22:25:57.403] > Waiting for server log... [22:25:57.436] > Waiting for server log... [22:25:57.470] > Waiting for server log... [22:25:57.503] > Waiting for server log... [22:25:57.538] > Waiting for server log... [22:25:57.572] > Waiting for server log... [22:25:57.605] > Waiting for server log... [22:25:57.640] > Waiting for server log... [22:25:57.674] > Waiting for server log... [22:25:57.708] > Waiting for server log... [22:25:57.743] > Waiting for server log... [22:25:57.775] > Waiting for server log... [22:25:57.810] > Waiting for server log... [22:25:57.848] > Waiting for server log... [22:25:57.879] > Waiting for server log... [22:25:57.913] > Waiting for server log... [22:25:57.948] > Waiting for server log... [22:25:57.982] > Waiting for server log... [22:25:58.016] > Waiting for server log... [22:25:58.050] > Waiting for server log... [22:25:58.083] > Waiting for server log... [22:25:58.117] > Waiting for server log... [22:25:58.152] > Waiting for server log... [22:25:58.186] > Waiting for server log... [22:25:58.220] > Waiting for server log... [22:25:58.254] > Waiting for server log... [22:25:58.287] > Waiting for server log... [22:25:58.322] > Waiting for server log... [22:25:58.356] > Waiting for server log... [22:25:58.390] > Waiting for server log... [22:25:58.424] > Waiting for server log... [22:25:58.458] > Waiting for server log... [22:25:58.492] > Waiting for server log... [22:25:58.526] > Waiting for server log... [22:25:58.561] > Waiting for server log... [22:25:58.594] > Waiting for server log... [22:25:58.629] > Waiting for server log... [22:25:58.664] > Waiting for server log... [22:25:58.700] > Waiting for server log... [22:25:58.734] > Waiting for server log... [22:25:58.769] > Waiting for server log... [22:25:58.802] > Waiting for server log... [22:25:58.837] > Waiting for server log... [22:25:58.871] > Waiting for server log... [22:25:58.904] > Waiting for server log... [22:25:58.938] > Waiting for server log... [22:25:58.973] > Waiting for server log... [22:25:59.006] > Waiting for server log... [22:25:59.040] > Waiting for server log... [22:25:59.074] > Waiting for server log... [22:25:59.108] > Waiting for server log... [22:25:59.142] > Waiting for server log... [22:25:59.176] > Waiting for server log... [22:25:59.209] > Waiting for server log... [22:25:59.243] > Waiting for server log... [22:25:59.278] > Waiting for server log... [22:25:59.312] > Waiting for server log... [22:25:59.345] > Waiting for server log... [22:25:59.380] > Waiting for server log... [22:25:59.413] > Waiting for server log... [22:25:59.446] > Waiting for server log... [22:25:59.482] > Waiting for server log... [22:25:59.514] > Waiting for server log... [22:25:59.549] > Waiting for server log... [22:25:59.583] > Waiting for server log... [22:25:59.616] > Waiting for server log... [22:25:59.650] > Waiting for server log... [22:25:59.692] > Waiting for server log... [22:25:59.719] > Waiting for server log... [22:25:59.753] > Waiting for server log... [22:25:59.788] > Waiting for server log... [22:25:59.822] > Waiting for server log... [22:25:59.856] > Waiting for server log... [22:25:59.890] > Waiting for server log... [22:25:59.924] > Waiting for server log... [22:25:59.958] > Waiting for server log... [22:25:59.991] > Waiting for server log... [22:26:00.026] > Waiting for server log... [22:26:00.059] > Waiting for server log... [22:26:00.093] > Waiting for server log... [22:26:00.128] > Waiting for server log... [22:26:00.161] > Waiting for server log... [22:26:00.195] > Waiting for server log... [22:26:00.229] > Waiting for server log... [22:26:00.263] > Waiting for server log... [22:26:00.298] > Waiting for server log... [22:26:00.332] > Waiting for server log... [22:26:00.366] > Waiting for server log... [22:26:00.402] > Waiting for server log... [22:26:00.436] > Waiting for server log... [22:26:00.469] > Waiting for server log... [22:26:00.503] > Waiting for server log... [22:26:00.537] > Waiting for server log... [22:26:00.570] > Waiting for server log... [22:26:00.604] > Waiting for server log... [22:26:00.638] > Waiting for server log... [22:26:00.679] > Waiting for server log... [22:26:00.705] > Waiting for server log... [22:26:00.740] > Waiting for server log... [22:26:00.773] > Waiting for server log... [22:26:00.807] > Waiting for server log... [22:26:00.841] > Waiting for server log... [22:26:00.875] > Waiting for server log... [22:26:00.909] > Waiting for server log... [22:26:00.943] > Waiting for server log... [22:26:00.977] > Waiting for server log... [22:26:01.010] > Waiting for server log... [22:26:01.045] > Waiting for server log... [22:26:01.078] > Waiting for server log... [22:26:01.112] > Waiting for server log... [22:26:01.147] > Waiting for server log... [22:26:01.179] > Waiting for server log... [22:26:01.214] > Waiting for server log... [22:26:01.249] > Waiting for server log... [22:26:01.282] > Waiting for server log... [22:26:01.316] > Waiting for server log... [22:26:01.350] > Waiting for server log... [22:26:01.383] > Waiting for server log... [22:26:01.418] > Waiting for server log... [22:26:01.453] > Waiting for server log... [22:26:01.486] > Waiting for server log... [22:26:01.520] > Waiting for server log... [22:26:01.554] > Waiting for server log... [22:26:01.588] > Waiting for server log... [22:26:01.621] > Waiting for server log... [22:26:01.656] > Waiting for server log... [22:26:01.689] > Waiting for server log... [22:26:01.723] > Waiting for server log... [22:26:01.758] > Waiting for server log... [22:26:01.791] > Waiting for server log... [22:26:01.825] > Waiting for server log... [22:26:01.860] > Waiting for server log... [22:26:01.894] > Waiting for server log... [22:26:01.928] > Waiting for server log... [22:26:01.962] > Waiting for server log... [22:26:01.995] > Waiting for server log... [22:26:02.030] > Waiting for server log... [22:26:02.064] > Waiting for server log... [22:26:02.097] > Waiting for server log... [22:26:02.130] > Waiting for server log... [22:26:02.165] > Waiting for server log... [22:26:02.199] > Waiting for server log... [22:26:02.232] > Waiting for server log... [22:26:02.267] > Waiting for server log... [22:26:02.300] > Waiting for server log... [22:26:02.334] > Waiting for server log... [22:26:02.369] > Waiting for server log... [22:26:02.402] > Waiting for server log... [22:26:02.436] > Waiting for server log... [22:26:02.471] > Waiting for server log... [22:26:02.504] > Waiting for server log... [22:26:02.538] > Waiting for server log... [22:26:02.572] > Waiting for server log... [22:26:02.605] > Waiting for server log... [22:26:02.639] > Waiting for server log... [22:26:02.674] > Waiting for server log... [22:26:02.708] > Waiting for server log... [22:26:02.741] > Waiting for server log... [22:26:02.776] > Waiting for server log... [22:26:02.809] > Waiting for server log... [22:26:02.843] > Waiting for server log... [22:26:02.878] > Waiting for server log... [22:26:02.911] > Waiting for server log... [22:26:02.945] > Waiting for server log... [22:26:02.983] > Waiting for server log... [22:26:03.014] > Waiting for server log... [22:26:03.049] > Waiting for server log... [22:26:03.084] > Waiting for server log... [22:26:03.116] > Waiting for server log... [22:26:03.151] > Waiting for server log... [22:26:03.185] > Waiting for server log... [22:26:03.218] > Waiting for server log... [22:26:03.252] > Waiting for server log... [22:26:03.287] > Waiting for server log... [22:26:03.321] > Waiting for server log... [22:26:03.354] > Waiting for server log... [22:26:03.389] > Waiting for server log... [22:26:03.423] > Waiting for server log... [22:26:03.456] > Waiting for server log... [22:26:03.491] > Waiting for server log... [22:26:03.525] > Waiting for server log... [22:26:03.558] > Waiting for server log... [22:26:03.601] > Waiting for server log... [22:26:03.627] > Waiting for server log... [22:26:03.661] > Waiting for server log... [22:26:03.696] > Waiting for server log... [22:26:03.730] > Waiting for server log... [22:26:03.764] > Waiting for server log... [22:26:03.799] > Waiting for server log... [22:26:03.832] > Waiting for server log... [22:26:03.866] > Waiting for server log... [22:26:03.900] > Waiting for server log... [22:26:03.934] > Waiting for server log... [22:26:03.969] > Waiting for server log... [22:26:04.002] > Waiting for server log... [22:26:04.036] > Waiting for server log... [22:26:04.070] > Waiting for server log... [22:26:04.104] > Waiting for server log... [22:26:04.137] > Waiting for server log... [22:26:04.171] > Waiting for server log... [22:26:04.206] > Waiting for server log... [22:26:04.239] > Waiting for server log... [22:26:04.273] > Waiting for server log... [22:26:04.308] > Waiting for server log... [22:26:04.342] > Waiting for server log... [22:26:04.376] > Waiting for server log... [22:26:04.411] > Waiting for server log... [22:26:04.445] > Waiting for server log... [22:26:04.479] > Waiting for server log... [22:26:04.514] > Waiting for server log... [22:26:04.548] > Waiting for server log... [22:26:04.581] > Waiting for server log... [22:26:04.616] > Waiting for server log... [22:26:04.649] > Waiting for server log... [22:26:04.683] > Waiting for server log... [22:26:04.717] > Waiting for server log... [22:26:04.752] > Waiting for server log... [22:26:04.787] > Waiting for server log... [22:26:04.820] > Waiting for server log... [22:26:04.854] > Waiting for server log... [22:26:04.887] > Waiting for server log... [22:26:04.921] > Waiting for server log... [22:26:04.955] > Waiting for server log... [22:26:04.989] > Waiting for server log... [22:26:05.024] > Waiting for server log... [22:26:05.057] > Waiting for server log... [22:26:05.091] > Waiting for server log... [22:26:05.125] > Waiting for server log... [22:26:05.159] > Waiting for server log... [22:26:05.192] > Waiting for server log... [22:26:05.227] > Waiting for server log... [22:26:05.260] > Waiting for server log... [22:26:05.294] > Waiting for server log... [22:26:05.329] > Waiting for server log... [22:26:05.363] > Waiting for server log... [22:26:05.396] > Waiting for server log... [22:26:05.432] > Waiting for server log... [22:26:05.465] > Waiting for server log... [22:26:05.500] > Waiting for server log... [22:26:05.534] > Waiting for server log... [22:26:05.567] > Waiting for server log... [22:26:05.601] > Waiting for server log... [22:26:05.636] > Waiting for server log... [22:26:05.670] > Waiting for server log... [22:26:05.704] > Waiting for server log... [22:26:05.738] > Waiting for server log... [22:26:05.772] > Waiting for server log... [22:26:05.806] > Waiting for server log... [22:26:05.840] > Waiting for server log... [22:26:05.874] > Waiting for server log... [22:26:05.908] > Waiting for server log... [22:26:05.942] > Waiting for server log... [22:26:05.975] > Waiting for server log... [22:26:06.009] > Waiting for server log... [22:26:06.043] > Waiting for server log... [22:26:06.077] > Waiting for server log... [22:26:06.111] > Waiting for server log... [22:26:06.145] > Waiting for server log... [22:26:06.179] > Waiting for server log... [22:26:06.214] > Waiting for server log... [22:26:06.249] > Waiting for server log... [22:26:06.282] > Waiting for server log... [22:26:06.316] > Waiting for server log... [22:26:06.352] > Waiting for server log... [22:26:06.385] > Waiting for server log... [22:26:06.419] > Waiting for server log... [22:26:06.454] > Waiting for server log... [22:26:06.488] > Waiting for server log... [22:26:06.521] > Waiting for server log... [22:26:06.555] > Waiting for server log... [22:26:06.590] > Waiting for server log... [22:26:06.623] > Waiting for server log... [22:26:06.657] > Waiting for server log... [22:26:06.692] > Waiting for server log... [22:26:06.725] > Waiting for server log... [22:26:06.759] > Waiting for server log... [22:26:06.794] > Waiting for server log... [22:26:06.827] > Waiting for server log... [22:26:06.861] > Waiting for server log... [22:26:06.895] > Waiting for server log... [22:26:06.929] > Waiting for server log... [22:26:06.963] > Waiting for server log... [22:26:06.998] > Waiting for server log... [22:26:07.031] > Waiting for server log... [22:26:07.065] > Waiting for server log... [22:26:07.099] > Waiting for server log... [22:26:07.133] > Waiting for server log... [22:26:07.167] > Waiting for server log... [22:26:07.202] > Waiting for server log... [22:26:07.236] > Waiting for server log... [22:26:07.270] > Waiting for server log... [22:26:07.305] > Waiting for server log... [22:26:07.337] > Waiting for server log... [22:26:07.372] > Waiting for server log... [22:26:07.406] > Waiting for server log... [22:26:07.439] > Waiting for server log... [22:26:07.473] > Waiting for server log... [22:26:07.508] > Waiting for server log... [22:26:07.541] > Waiting for server log... [22:26:07.575] > Waiting for server log... [22:26:07.609] > Waiting for server log... [22:26:07.643] > Waiting for server log... [22:26:07.677] > Waiting for server log... [22:26:07.711] > Waiting for server log... [22:26:07.745] > Waiting for server log... [22:26:07.778] > Waiting for server log... [22:26:07.813] > Waiting for server log... [22:26:07.846] > Waiting for server log... [22:26:07.881] > Waiting for server log... [22:26:07.915] > Waiting for server log... [22:26:07.949] > Waiting for server log... [22:26:07.982] > Waiting for server log... [22:26:08.018] > > * [22:26:08.019] > * Visual Studio Code Server > * > * By using the software, you agree to > * the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and > * the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement). > * > [22:26:08.023] > Server did not start successfully. Full server log at /home/users/keyon/.vscode-server/.441438abd1ac652551dbe4d408dfcec8a499b8bf.log >>> [22:26:08.025] > <<< End of server log [22:26:08.026] > e5ab059fc972: start > exitCode==32== > listeningOn==== > osReleaseId==centos== > arch==x86_64== > tmpDir==/tmp== > platform==linux== > unpackResult==success== > didLocalDownload==0== > downloadTime==859== > installTime==12195== > extInstallTime==== > serverStartTime==15011== > e5ab059fc972: end [22:26:08.026] Received install output: exitCode==32== listeningOn==== osReleaseId==centos== arch==x86_64== tmpDir==/tmp== platform==linux== unpackResult==success== didLocalDownload==0== downloadTime==859== installTime==12195== extInstallTime==== serverStartTime==15011== [22:26:08.027] Terminating local server [22:26:08.030] Resolver error: Error: The VS Code Server failed to start at g.ServerInstallError (/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:584081) at f (/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:578291) at t.handleInstallOutput (/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:581316) at Object.e [as tryInstallWithLocalServer] (/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:623419) at process.processTicksAndRejections (node:internal/process/task_queues:96:5) at async /Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:642712 at async t.withShowDetailsEvent (/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:646432) at async /Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:621891 at async T (/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:618397) at async t.resolveWithLocalServer (/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:621506) at async t.resolve (/Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:644040) at async /Users/keyonvafa/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/extension.js:1:721750 [22:26:08.033] Local server exit: 15 [22:26:08.039] ------ ```

Anything else?

Thank you!

eleanorjboyd commented 1 year ago

Hello! I few questions/ comments

  1. Can you include the full remote-ssh logs?
  2. Do you have a windows computer / recently updated?
  3. Can you ask send your command you tried in the "connect locally" second of this form.

Usually when it doesn't work to connect locally with the same connection string we are using it is due to an issue with your setup and not with the extension (since it doesn't work from the command line). This being said I might have some advice on maybe a setting to change. If you could try removing different args from the connection command that might help you narrow down which command line arg is causing the problem. So if the string is "ssh localhost -D -t" trying it without the -D and -T respectively to see if one of those is the issue.

keyonvafa commented 1 year ago

Thank you for the response.

  1. Weirdly there is no log available... the output says Full server log at /home/users/keyon/.vscode-server/.f80445acd5a3dadef24aa209168452a3d97cc326.log, and this file exists, but it is empty. I've updated my original post with the full output from the output tab.
  2. No -- I have a Mac computer and I am connecting to a Linux system
  3. Maybe I misunderstood the connect locally section of the form -- I just tried toggling "remote.SSH.useLocalServer" between true and false. In both settings, I didn't input a specific SSH command -- I used Remote to connect to the host as usual, and both times the same error happened. When I connect in the terminal outside of VSCode it works fine.
eleanorjboyd commented 1 year ago

By the whole logs do you mean the output tab and then selecting "remote-ssh" from the dropdown? We print out more to our logs that what it seems like you have (like there should be other print out before [20:45:58.199] > ready: 5f3a604a11fc). Specifically, we print out a line in the logs that says Running ssh connection command and this is the one which will provide the best help as it prints exactly what we want to run locally in a terminal outside vscode to replicate the behavior.

keyonvafa commented 1 year ago

Ah yes sorry I wasn't including the full log -- I just edited it to include the part before.

eleanorjboyd commented 1 year ago

Hi! So it looks like you are back a few versions on the remote-ssh extension- does upgrading do anything? And the log string I am asking for is not in that version of the extension but they are still listed so if you could run this and see the option.

ssh -v -T -D 56207 -o ConnectTimeout=30 sh03-ln06.stanford.edu

so following these steps using that command string:

Usually when it doesn't work to connect locally with the same connection string we are using it is due to an issue with your setup and not with the extension (since it doesn't work from the command line). This being said I might have some advice on maybe a setting to change. If you could try removing different args from the connection command that might help you narrow down which command line arg is causing the problem. So if the string is "ssh localhost -D -t" trying it without the -D and -T respectively to see if one of those is the issue.

keyonvafa commented 1 year ago

I just installed the latest version of VSCode and the newest Remote-SSH. The same error happens, I've updated the log above.

Thanks for including the ssh command -- as written, the connection hangs, but when I drop the -T it works.

eleanorjboyd commented 1 year ago

Ok great thank you for finding this. This is a known bug that some people are running into- you can see it here as well: https://github.com/microsoft/vscode-remote-release/issues/7673. I have created an issue to fix it here: https://github.com/microsoft/vscode-remote-release/issues/7558. If you could go and upvote that issue it would be helpful for keeping track of how many people are affected and prioritization.

Also, could you outline any guesses you have about what might have changed between it working and not? We have always included the -T arg so there must be a regression and I am still unsure of where (could be openSSH, computer update etc). Thanks

keyonvafa commented 1 year ago

Got it thanks! Upvoted and I'll be following the issue.

It's weird, I don't have a good sense of what may be causing the issue. I want to say I recently updated VS Code but now when I uninstall and use older versions that used to work I have the same issue. I also regularly use VS Code to log into two clusters but the issue only occurs for one of the clusters, and the other cluster works fine. Moreover I asked one other person who uses the problematic cluster to see if they were also having this issue and they reported the same problem.

keyonvafa commented 1 year ago

Also looking at #7673 I should specify that I'm not running OS Ventura (I'm still on Monterey) but I'm running into the issue.

eleanorjboyd commented 1 year ago

Thank you for the feedback! Do you know any differences between the two clusters?

keyonvafa commented 1 year ago

I don't have much insight into how they differ under the hood. But one weird thing is that the behavior with and without using the -T flag to SSH in using the command you sent is the same for both clusters even though Remote works for one of them and it doesn't work for the other.

And even when using the -T flag, there is some functionality of the clusters, so I'm not sure it's the same segfault that I'm getting on VSCode. Here I am showing the difference between using the -T flag and not. When the -T flag is included, the output is weirdly formatted as the prompt doesn't show up and I can't really edit files. When the -T flag is dropped, everything works as desired. And the behavior for both of the clusters is the same with respect to the -T flag, so I'm not sure if that's the main source of the issue...

With the -T flag:

Screenshot 2023-02-21 at 10 08 32 PM

Without the -T flag:

Screenshot 2023-02-21 at 10 13 45 PM
cffan commented 1 year ago

Same problem here, and I'm on the same cluster as @keyonvafa.

I think the root cause is not -T, but the following line in the log he uploaded:

[22:25:53.112] stderr> bash: line 504: 92477 Segmentation fault      $VSCH_SERVER_SCRIPT --start-server --host=127.0.0.1 --accept-server-license-terms --enable-remote-auto-shutdown $PORT_FLAG $TELEMETRY $SERVER_LISTEN_FLAG $EXTENSIONS --connection-token-file $VSCH_TOKENFILE &>"$VSCH_LOGFILE" < /dev/null

I saw the same seg fault message in my vscode ssh log. After some digging, I found the problem very intriguing.. The VSCH_SERVER_SCRIPT script launches a node server. If I manually call the node binary with relative path like this ./node, it launches fine. However, if call it with absolute path ~/.vscode-server/bin/74b1f979648cc44d385a2286793c226e611f59e7/node, it will seg fault. strace doesn't give any useful information.

I'm clueless at this point...

zyzhang1992 commented 1 year ago

I once got a segfault launching the server on one of the log in node but not on the compute node. I mostly launched the server on compute node. I was thinking running the server may be taking too much memory on the log in node but @cffan's test show it may not be a memory issue. It is more likely a bug exposed only when the program name becomes long.

cffan commented 1 year ago

I once got a segfault launching the server on one of the log in node but not on the compute node. I mostly launched the server on compute node. I was thinking running the server may be taking too much memory on the log in node but @cffan's test show it may not be a memory issue. It is more likely a bug exposed only when the program name becomes long.

@zyzhang1992 How do you run vscode in compute node? Does that requires some additional ssh tunneling setup?

zyzhang1992 commented 1 year ago

@eleanorjboyd I think I can confirm that the problem is specific to the log in node. We access the cluster through the log in node and then allocate resources on the compute nodes. The log in nodes use cgroup controls of the resources. There may also be limitations to the stack/heap and file meta data on the log in nodes. Look like that there might be a bug of remote-ssh that are only exposed when the server is launched on the log in nodes. Let us know how we can help resolving this issue. Thanks!

cffan commented 1 year ago

@zyzhang1992 @keyonvafa just FYI, my current work around is to use remote tunnel on the compute node.

zyzhang1992 commented 1 year ago

@cffan how did you set it up? Do you log on to a compute node and then start the server on the compute node? That is something I was hoping to do but haven't had the time to try that yet.

cffan commented 1 year ago

@cffan how did you set it up? Do you log on to a compute node and then start the server on the compute node? That is something I was hoping to do but haven't had the time to try that yet.

Yes. I use code tunnel in compute node and then follow the instructions.

zyzhang1992 commented 1 year ago

Host sherlock HostName sh02-ln02.stanford.edu ForwardX11Trusted yes ForwardX11 yes User zyzhang

Host sherlock-compute ForwardX11Trusted yes ForwardX11 yes User zyzhang ProxyCommand ssh sherlock "nc $(squeue --me --name=vscode-tunnel -h -O Nodelist) 22"

@cffan here is the config I use for lauching the server on the compute node. I have a slurm job with the job name vscode-tunnel for the allocation. I can go over it with you during the OH if you think that may help.

eleanorjboyd commented 1 year ago

Thank you for the additional information and context. I am going to close this issue now as it seems we have reached an endpoint in this discussion but please reopen if that is not the case. Thanks

neergaard commented 1 year ago

I have been having this issue for months now also on the same cluster as @cffan and @keyonvafa and it's also an issue with the login node, and I would very much like a solution.

I like having the possibility of connecting to the login node and then connect to interactive sessions from there, so if I can avoid connecting directly to the compute nodes, that would be great.

I have a log file that also points to a segfault, but I initially thought if it could be caused by a libstdc++ version mismatch. vscode_remote_log.txt

zyzhang1992 commented 1 year ago

that is unlikely since it is the same libraries on both the log in and the compute nodes. segfault only occurs on the log in node, not the compute node.


From: Alexander Neergaard Zahid @.> Sent: Wednesday, March 8, 2023 12:36 PM To: microsoft/vscode-remote-release @.> Cc: Zhiyong Zhang @.>; Mention @.> Subject: Re: [microsoft/vscode-remote-release] [Remote-SSH Bug]: Remote SSH fails at "Waiting for server log..." (Issue #8063)

I have been having this issue for months now also on the same cluster as @cffanhttps://github.com/cffan and @keyonvafahttps://github.com/keyonvafa and it's also an issue with the login node.

I have a log file that also points to a segfault, but I was wondering if it could be caused by a libstdc++ version mismatch? vscode_remote_log.txthttps://github.com/microsoft/vscode-remote-release/files/10924987/vscode_remote_log.txt

See herehttps://code.visualstudio.com/docs/remote/linux#_remote-host-container-wsl-linux-prerequisites

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/vscode-remote-release/issues/8063#issuecomment-1460844043, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADG474FIJIUBEGJOSOTAS6DW3DUUXANCNFSM6AAAAAAVAQP7W4. You are receiving this because you were mentioned.Message ID: @.***>