[Remote-SSH Bug]: Remote SSH fails at "Waiting for server log..." after connection dropout

Is there an existing issue for this bug?

[X] I have searched the existing issues

Required Troubleshooting Steps

[X] I have followed these troubleshooting steps
[X] I have tried both values of the remote.SSH.useLocalServer setting

Connect Locally

It doesn’t connect successfully (different error message)

->

Below is the code & output. After all this, there was no command prompt so I'm assuming the login nearly worked, but failed at the last step. My only option to regain control of my terminal was Crtl+c

$ ssh -v -T -D 44835 -o ConnectTimeout=15 phoenix
OpenSSH_8.2p1 Ubuntu-4ubuntu0.11, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/steviep/.ssh/config
debug1: /home/steviep/.ssh/config line 29: Applying options for phoenix
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to p2-log-1.hpc.adelaide.edu.au [10.34.33.1] port 22.
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: identity file /home/steviep/.ssh/id_rsa type -1
debug1: identity file /home/steviep/.ssh/id_rsa-cert type -1
debug1: identity file /home/steviep/.ssh/id_dsa type -1
debug1: identity file /home/steviep/.ssh/id_dsa-cert type -1
debug1: identity file /home/steviep/.ssh/id_ecdsa type -1
debug1: identity file /home/steviep/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/steviep/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/steviep/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/steviep/.ssh/id_ed25519 type 3
debug1: identity file /home/steviep/.ssh/id_ed25519-cert type -1
debug1: identity file /home/steviep/.ssh/id_ed25519_sk type -1
debug1: identity file /home/steviep/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/steviep/.ssh/id_xmss type -1
debug1: identity file /home/steviep/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.11
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to p2-log-1.hpc.adelaide.edu.au:22 as 'a1018048'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:R9IY5h4bjTgb4djJQ6G2qQ92s3JQEvVkAfNX2Rc9x2A
debug1: Host 'p2-log-1.hpc.adelaide.edu.au' is known and matches the ECDSA host key.
debug1: Found key in /home/steviep/.ssh/known_hosts:17
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/steviep/.ssh/id_ed25519 ED25519 SHA256:uk2unnijAXKsn7xkJ5RUJGcvsYBfcaYYKSbTnokwCGU agent
debug1: Will attempt key: /home/steviep/.ssh/id_rsa 
debug1: Will attempt key: /home/steviep/.ssh/id_dsa 
debug1: Will attempt key: /home/steviep/.ssh/id_ecdsa 
debug1: Will attempt key: /home/steviep/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/steviep/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/steviep/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: Next authentication method: publickey
debug1: Offering public key: /home/steviep/.ssh/id_ed25519 ED25519 SHA256:uk2unnijAXKsn7xkJ5RUJGcvsYBfcaYYKSbTnokwCGU agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/steviep/.ssh/id_rsa
debug1: Trying private key: /home/steviep/.ssh/id_dsa
debug1: Trying private key: /home/steviep/.ssh/id_ecdsa
debug1: Trying private key: /home/steviep/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/steviep/.ssh/id_ed25519_sk
debug1: Trying private key: /home/steviep/.ssh/id_xmss
debug1: Next authentication method: password
a1018048@p2-log-1.hpc.adelaide.edu.au's password: 
debug1: Authentication succeeded (password).
Authenticated to p2-log-1.hpc.adelaide.edu.au ([10.34.33.1]:22).
debug1: Local connections to LOCALHOST:44835 forwarded to remote address socks:0
debug1: Local forwarding listening on ::1 port 44835.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 44835.
debug1: channel 1: new [port listener]
debug1: channel 2: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_AU.UTF-8
Last failed login: Mon Jan 29 23:27:48 ACDT 2024 from 10.210.32.82 on ssh:notty
There were 4 failed login attempts since the last successful login.

Expected Behavior

Today, my internet dropped out whilst connected to a remote server and I had to close VSCode whilst the connection was hung. Since then, I have not been able to login to that server. The behaviour seems highly reminiscent of #8063 .

I have also tried removing ~/.vscode-server on the server but get the following output

$ cd ~
$ rm -rf .vscode-server
rm: cannot remove '.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/.nfs00000001c63bca0c000016d9': Device or resource busy
rm: cannot remove '.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/node_modules/vsda/build/Release': Directory not empty
rm: cannot remove '.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/node_modules/@vscode/spdlog/build/Release': Directory not empty
rm: cannot remove '.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/.nfs00000001c77a78ca0000110a': Device or resource busy
rm: cannot remove '.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/bin': Directory not empty
rm: cannot remove '.vscode-server/data/logs/20240129T120720/exthost1/vscode.git': Directory not empty
rm: cannot remove '.vscode-server/data/logs/20240129T120720/exthost1/output_logging_20240129T120721': Directory not empty
rm: cannot remove '.vscode-server/data/logs/20240129T120720/exthost1/vscode.github': Directory not empty
rm: cannot remove '.vscode-server/data/logs/20240124T122317': Directory not empty

I can ssh into the remote using the generic command ssh phoenix and whilst logged in have tried the following without success

ps aux | grep .vscode-server | awk '{print $2}' | xargs kill

I have also tried killing individual processes without success

I have also tried using 1) Remote-SSH: kill VS Code Server on Host... and 2) Remote-SSH: Uninstall VS Code Server on Host... with no change to the behaviour

Steps To Reproduce

I can connect to other servers without problem and it appears the problem is unique to this specific server, probably caused by the dropout

Remote-SSH Log

``` [23:26:22.241] Log Level: 2 [23:26:22.249] VS Code version: 1.85.2 [23:26:22.249] Remote-SSH version: remote-ssh@0.107.1 [23:26:22.249] linux x64 [23:26:22.250] SSH Resolver called for "ssh-remote+phoenix", attempt 1 [23:26:22.251] "remote.SSH.useLocalServer": true [23:26:22.251] "remote.SSH.useExecServer": false [23:26:22.251] "remote.SSH.path": undefined [23:26:22.251] "remote.SSH.configFile": undefined [23:26:22.251] "remote.SSH.useFlock": true [23:26:22.251] "remote.SSH.lockfilesInTmp": false [23:26:22.251] "remote.SSH.localServerDownload": auto [23:26:22.251] "remote.SSH.remoteServerListenOnSocket": false [23:26:22.251] "remote.SSH.showLoginTerminal": false [23:26:22.252] "remote.SSH.defaultExtensions": [] [23:26:22.252] "remote.SSH.loglevel": 2 [23:26:22.252] "remote.SSH.enableDynamicForwarding": true [23:26:22.252] "remote.SSH.enableRemoteCommand": false [23:26:22.252] "remote.SSH.serverPickPortsFromRange": {} [23:26:22.252] "remote.SSH.serverInstallPath": {} [23:26:22.258] SSH Resolver called for host: phoenix [23:26:22.258] Setting up SSH remote "phoenix" [23:26:22.260] Acquiring local install lock: /tmp/vscode-remote-ssh-74cd3cec-install.lock [23:26:22.260] Looking for existing server data file at /home/steviep/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-74cd3cec-8b3775030ed1a69b13e4f4c628c612102e30a681-0.107.1-tr/data.json [23:26:22.261] Using commit id "8b3775030ed1a69b13e4f4c628c612102e30a681" and quality "stable" for server [23:26:22.263] Install and start server if needed [23:26:22.264] PATH: /home/steviep/mambaforge/bin:/home/steviep/mambaforge/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin [23:26:22.264] Checking ssh with "ssh -V" [23:26:22.270] > OpenSSH_8.2p1 Ubuntu-4ubuntu0.11, OpenSSL 1.1.1f 31 Mar 2020 [23:26:22.274] askpass server listening on /run/user/1000/vscode-ssh-askpass-e5ae3a8610e6d60d5f17c609af436948fc1b5725.sock [23:26:22.274] Spawning local server with {"serverId":1,"ipcHandlePath":"/run/user/1000/vscode-ssh-askpass-5bda574219dc5ec39a211896d9c73f687911fd8c.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","45045","-o","ConnectTimeout=15","phoenix"],"serverDataFolderName":".vscode-server","dataFilePath":"/home/steviep/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-74cd3cec-8b3775030ed1a69b13e4f4c628c612102e30a681-0.107.1-tr/data.json"} [23:26:22.274] Local server env: {"SSH_AUTH_SOCK":"/run/user/1000/keyring/ssh","SHELL":"/bin/bash","DISPLAY":":0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/usr/share/code/code","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/run/user/1000/vscode-ssh-askpass-e5ae3a8610e6d60d5f17c609af436948fc1b5725.sock"} [23:26:22.277] Spawned 6156 [23:26:22.357] > local-server-1> Running ssh connection command: "-v -T -D 45045 -o ConnectTimeout=15 phoenix" [23:26:22.361] > local-server-1> Spawned ssh, pid=6165 [23:26:22.363] stderr> OpenSSH_8.2p1 Ubuntu-4ubuntu0.11, OpenSSL 1.1.1f 31 Mar 2020 [23:26:22.465] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:R9IY5h4bjTgb4djJQ6G2qQ92s3JQEvVkAfNX2Rc9x2A [23:26:22.605] stderr> No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000) [23:26:22.605] stderr> [23:26:22.605] stderr> [23:26:22.642] stderr> No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000) [23:26:22.642] stderr> [23:26:22.642] stderr> [23:26:22.749] Got askpass request: {"request":"a1018048@p2-log-1.hpc.adelaide.edu.au's password:"} [23:26:22.749] Showing password prompt [23:26:22.750] Listening for interwindow password on /run/user/1000/vscode-ssh-askpass-ce850c8e3acbc33cca2836235e8c4791d026361c.sock [23:26:22.750] Writing password prompt to globalState [23:26:29.914] Got password response [23:26:29.915] Interactor gave response: ************ [23:26:29.915] Cleaning up other-window auth server [23:26:29.985] stderr> Authenticated to p2-log-1.hpc.adelaide.edu.au ([10.34.33.1]:22). [23:26:30.108] > Last failed login: Mon Jan 29 23:16:15 ACDT 2024 from 10.210.32.82 on ssh:notty > There were 2 failed login attempts since the last successful login. [23:26:30.331] > ready: cf1ea10c7283 [23:26:30.350] > Linux 4.18.0-305.el8.x86_64 #1 SMP Thu Apr 29 08:54:30 EDT 2021 [23:26:30.350] Platform: linux [23:26:30.367] > /bin/bash [23:26:30.367] Shell: bash [23:26:30.367] Creating bash subshell inside "bash" [23:26:30.385] > bash version: 4.4.19(1)-release [23:26:30.431] > cf1ea10c7283: running [23:26:30.448] > Acquiring lock on /home/a1018048/.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/vscode-remote-lock.a1018048.8b3775030ed1a69b13e4f4c628c612102e30a681 [23:26:30.451] > Found existing installation at /home/a1018048/.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681... > Checking /home/a1018048/.vscode-server/.8b3775030ed1a69b13e4f4c628c612102e30a681.log and /home/a1018048/.vscode-server/.8b3775030ed1a69b13e4f4c628c612102e30a681.pid for a running server [23:26:30.454] > Looking for server with pid: 2179358 [23:26:30.508] > Running ssh connection command... /home/a1018048/.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/bin/code-server --start-server --host=127.0.0.1 --accept-server-license-terms --enable-remote-auto-shutdown --port=0 --telemetry-level all &> "/home/a1018048/.vscode-server/.8b3775030ed1a69b13e4f4c628c612102e30a681.log" < /dev/null > printenv: [23:26:30.542] > SSH_CONNECTION=10.210.32.82 48278 10.34.33.1 22 > LANG=en_AU.UTF-8 > HISTCONTROL=ignoredups > HOSTNAME=p2-log-1 > S_COLORS=auto > XDG_SESSION_ID=13600 > USER=a1018048 > __LMOD_REF_COUNT_MODULEPATH=/opt/ohpc/pub/modulefiles:1;/gpfs/apps/icl/modules/all:1 > PWD=/home/a1018048 > HOME=/home/a1018048 > LMOD_COLORIZE=no > SSH_CLIENT=10.210.32.82 48278 22 > LMOD_VERSION=8.2.10 > LMOD_SETTARG_CMD=: > BASH_ENV=/opt/ohpc/admin/lmod/lmod/init/bash > _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXt9LG1wYXRoQT17Ii9vcHQvb2hwYy9wdWIvbW9kdWxlZmlsZXMiLCIvZ3Bmcy9hcHBzL2ljbC9tb2R1bGVzL2FsbCIsfSxbInN5c3RlbUJhc2VNUEFUSCJdPSIvb3B0L29ocGMvcHViL21vZHVsZWZpbGVzOi9ncGZzL2FwcHMvaWNsL21vZHVsZXMvYWxsIix9 > LMOD_ROOT=/opt/ohpc/admin/lmod > MAIL=/var/spool/mail/a1018048 > SHELL=/bin/bash > _ModuleTable_Sz_=1 > SHLVL=2 > MANPATH=/opt/confluent/share/man: > VSCODE_AGENT_FOLDER=/home/a1018048/.vscode-server > LMOD_PREPEND_BLOCK=normal > MODULEPATH=/opt/ohpc/pub/modulefiles:/gpfs/apps/icl/modules/all > LOGNAME=a1018048 > XDG_RUNTIME_DIR=/run/user/1018048 > PATH=/opt/lenovo/onecli:/opt/confluent/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin > MODULESHOME=/opt/ohpc/admin/lmod/lmod > LMOD_SETTARG_FULL_SUPPORT=no > HISTSIZE=1000 > LMOD_PKG=/opt/ohpc/admin/lmod/lmod > LMOD_CMD=/opt/ohpc/admin/lmod/lmod/libexec/lmod > LESSOPEN=||/usr/bin/lesspipe.sh %s > LMOD_FULL_SETTARG_SUPPORT=no > LMOD_DIR=/opt/ohpc/admin/lmod/lmod/libexec > BASH_FUNC_module%%=() { eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh) > } > BASH_FUNC_ml%%=() { eval $($LMOD_DIR/ml_cmd "$@") > } > _=/usr/bin/printenv > Removing old logfile at /home/a1018048/.vscode-server/.8b3775030ed1a69b13e4f4c628c612102e30a681.log > Spawned remote server: 2183557 > Waiting for server log... [23:26:30.562] > Waiting for server log... [23:26:30.595] > Waiting for server log... [23:26:30.627] > Waiting for server log... [23:26:30.659] > Waiting for server log... [23:26:30.691] > Waiting for server log... [23:26:30.724] > Waiting for server log... [23:26:30.755] > Waiting for server log... [23:26:30.837] > Waiting for server log... > Waiting for server log... [23:26:30.853] > Waiting for server log... [23:26:30.885] > Waiting for server log... [23:26:30.917] > Waiting for server log... [23:26:30.949] > Waiting for server log... [23:26:30.982] > Waiting for server log... [23:26:31.014] > Waiting for server log... [23:26:31.046] > Waiting for server log... [23:26:31.079] > Waiting for server log... [23:26:31.110] > Waiting for server log... [23:26:31.143] > Waiting for server log... [23:26:31.175] > Waiting for server log... [23:26:31.210] > Waiting for server log... [23:26:31.239] > Waiting for server log... [23:26:31.272] > Waiting for server log... [23:26:31.352] > Waiting for server log... > Waiting for server log... [23:26:31.369] > Waiting for server log... [23:26:31.401] > Waiting for server log... [23:26:31.433] > Waiting for server log... [23:26:31.514] > Waiting for server log... > Waiting for server log... [23:26:31.531] > Waiting for server log... [23:26:31.563] > Waiting for server log... [23:26:31.595] > Waiting for server log... [23:26:31.627] > Waiting for server log... [23:26:31.659] > Waiting for server log... [23:26:31.690] > Waiting for server log... [23:26:31.724] > Waiting for server log... [23:26:31.756] > Waiting for server log... [23:26:31.787] > Waiting for server log... [23:26:31.820] > Waiting for server log... [23:26:31.852] > Waiting for server log... [23:26:31.884] > Waiting for server log... [23:26:31.916] > Waiting for server log... [23:26:31.949] > Waiting for server log... [23:26:31.982] > Waiting for server log... [23:26:32.014] > Waiting for server log... [23:26:32.047] > Waiting for server log... [23:26:32.079] > Waiting for server log... [23:26:32.110] > Waiting for server log... [23:26:32.142] > Waiting for server log... [23:26:32.174] > Waiting for server log... [23:26:32.206] > Waiting for server log... [23:26:32.239] > Waiting for server log... [23:26:32.272] > Waiting for server log... [23:26:32.308] > Waiting for server log... [23:26:32.336] > Waiting for server log... [23:26:32.369] > Waiting for server log... [23:26:32.400] > Waiting for server log... [23:26:32.432] > Waiting for server log... [23:26:32.464] > Waiting for server log... [23:26:32.496] > Waiting for server log... [23:26:32.529] > Waiting for server log... [23:26:32.561] > Waiting for server log... [23:26:32.594] > Waiting for server log... [23:26:32.626] > Waiting for server log... [23:26:32.657] > Waiting for server log... [23:26:32.690] > Waiting for server log... [23:26:32.723] > Waiting for server log... [23:26:32.754] > Waiting for server log... [23:26:32.786] > Waiting for server log... [23:26:32.819] > Waiting for server log... [23:26:32.851] > Waiting for server log... [23:26:32.883] > Waiting for server log... [23:26:32.916] > Waiting for server log... [23:26:32.947] > Waiting for server log... [23:26:32.980] > Waiting for server log... [23:26:33.012] > Waiting for server log... [23:26:33.044] > Waiting for server log... [23:26:33.076] > Waiting for server log... [23:26:33.109] > Waiting for server log... [23:26:33.140] > Waiting for server log... [23:26:33.172] > Waiting for server log... [23:26:33.207] > Waiting for server log... [23:26:33.237] > Waiting for server log... [23:26:33.269] > Waiting for server log... [23:26:33.302] > Waiting for server log... [23:26:33.334] > Waiting for server log... [23:26:33.367] > Waiting for server log... [23:26:33.398] > Waiting for server log... [23:26:33.430] > Waiting for server log... [23:26:33.462] > Waiting for server log... [23:26:33.494] > Waiting for server log... [23:26:33.527] > Waiting for server log... [23:26:33.559] > Waiting for server log... [23:26:33.591] > Waiting for server log... [23:26:33.623] > Waiting for server log... [23:26:33.655] > Waiting for server log... [23:26:33.688] > Waiting for server log... [23:26:33.720] > Waiting for server log... [23:26:33.752] > Waiting for server log... [23:26:33.784] > Waiting for server log... [23:26:33.816] > Waiting for server log... [23:26:33.849] > Waiting for server log... [23:26:33.880] > Waiting for server log... [23:26:33.913] > Waiting for server log... [23:26:33.945] > Waiting for server log... [23:26:33.978] > Waiting for server log... [23:26:34.009] > Waiting for server log... [23:26:34.041] > Waiting for server log... [23:26:34.074] > Waiting for server log... [23:26:34.106] > Waiting for server log... [23:26:34.138] > Waiting for server log... [23:26:34.171] > Waiting for server log... [23:26:34.203] > Waiting for server log... [23:26:34.235] > Waiting for server log... [23:26:34.268] > Waiting for server log... [23:26:34.300] > Waiting for server log... [23:26:34.332] > Waiting for server log... [23:26:34.413] > Waiting for server log... > Waiting for server log... [23:26:34.429] > Waiting for server log... [23:26:34.461] > Waiting for server log... [23:26:34.492] > Waiting for server log... [23:26:34.525] > Waiting for server log... [23:26:34.557] > Waiting for server log... [23:26:34.589] > Waiting for server log... [23:26:34.622] > Waiting for server log... [23:26:34.654] > Waiting for server log... [23:26:34.686] > Waiting for server log... [23:26:34.718] > Waiting for server log... [23:26:34.750] > Waiting for server log... [23:26:34.783] > Waiting for server log... [23:26:34.815] > Waiting for server log... [23:26:34.896] > Waiting for server log... > Waiting for server log... [23:26:34.915] > Waiting for server log... [23:26:34.944] > Waiting for server log... [23:26:34.978] > Waiting for server log... [23:26:35.008] > Waiting for server log... [23:26:35.040] > Waiting for server log... [23:26:35.072] > Waiting for server log... [23:26:35.105] > Waiting for server log... [23:26:35.137] > Waiting for server log... [23:26:35.170] > Waiting for server log... [23:26:35.202] > Waiting for server log... [23:26:35.282] > Waiting for server log... > Waiting for server log... [23:26:35.299] > Waiting for server log... [23:26:35.331] > Waiting for server log... [23:26:35.363] > Waiting for server log... [23:26:35.395] > Waiting for server log... [23:26:35.427] > Waiting for server log... [23:26:35.460] > Waiting for server log... [23:26:35.491] > Waiting for server log... [23:26:35.523] > Waiting for server log... [23:26:35.606] > Waiting for server log... > Waiting for server log... [23:26:35.623] > Waiting for server log... [23:26:35.653] > Waiting for server log... [23:26:35.685] > Waiting for server log... [23:26:35.718] > Waiting for server log... [23:26:35.749] > Waiting for server log... [23:26:35.782] > Waiting for server log... [23:26:35.814] > Waiting for server log... [23:26:35.846] > Waiting for server log... [23:26:35.879] > Waiting for server log... [23:26:35.910] > Waiting for server log... [23:26:35.942] > Waiting for server log... [23:26:35.976] > Waiting for server log... [23:26:36.007] > Waiting for server log... [23:26:36.039] > Waiting for server log... [23:26:36.072] > Waiting for server log... [23:26:36.103] > Waiting for server log... [23:26:36.184] > Waiting for server log... > Waiting for server log... [23:26:36.204] > Waiting for server log... [23:26:36.281] > Waiting for server log... > Waiting for server log... [23:26:36.297] > Waiting for server log... [23:26:36.329] > Waiting for server log... [23:26:36.362] > Waiting for server log... [23:26:36.393] > Waiting for server log... [23:26:36.476] > Waiting for server log... > Waiting for server log... [23:26:36.493] > Waiting for server log... [23:26:36.522] > Waiting for server log... [23:26:36.553] > Waiting for server log... [23:26:36.586] > Waiting for server log... [23:26:36.619] > Waiting for server log... [23:26:36.650] > Waiting for server log... [23:26:36.683] > Waiting for server log... [23:26:36.715] > Waiting for server log... [23:26:36.748] > Waiting for server log... [23:26:36.780] > Waiting for server log... [23:26:36.812] > Waiting for server log... [23:26:36.844] > Waiting for server log... [23:26:36.876] > Waiting for server log... [23:26:36.909] > Waiting for server log... [23:26:36.941] > Waiting for server log... [23:26:36.973] > Waiting for server log... [23:26:37.005] > Waiting for server log... [23:26:37.038] > Waiting for server log... [23:26:37.069] > Waiting for server log... [23:26:37.102] > Waiting for server log... [23:26:37.134] > Waiting for server log... [23:26:37.166] > Waiting for server log... [23:26:37.203] > Waiting for server log... [23:26:37.231] > Waiting for server log... [23:26:37.262] > Waiting for server log... [23:26:37.294] > Waiting for server log... [23:26:37.326] > Waiting for server log... [23:26:37.359] > Waiting for server log... [23:26:37.391] > Waiting for server log... [23:26:37.423] > Waiting for server log... [23:26:37.455] > Waiting for server log... [23:26:37.487] > Waiting for server log... [23:26:37.519] > Waiting for server log... [23:26:37.552] > Waiting for server log... [23:26:37.584] > Waiting for server log... [23:26:37.665] > Waiting for server log... > Waiting for server log... [23:26:37.681] > Waiting for server log... [23:26:37.714] > Waiting for server log... [23:26:37.746] > Waiting for server log... [23:26:37.779] > Waiting for server log... [23:26:37.810] > Waiting for server log... [23:26:37.891] > Waiting for server log... > Waiting for server log... [23:26:37.907] > Waiting for server log... [23:26:37.939] > Waiting for server log... [23:26:37.971] > Waiting for server log... [23:26:38.003] > Waiting for server log... [23:26:38.035] > Waiting for server log... [23:26:38.115] > Waiting for server log... > Waiting for server log... [23:26:38.133] > Waiting for server log... [23:26:38.165] > Waiting for server log... [23:26:38.197] > Waiting for server log... [23:26:38.229] > Waiting for server log... [23:26:38.261] > Waiting for server log... [23:26:38.294] > Waiting for server log... [23:26:38.325] > Waiting for server log... [23:26:38.357] > Waiting for server log... [23:26:38.390] > Waiting for server log... [23:26:38.422] > Waiting for server log... [23:26:38.454] > Waiting for server log... [23:26:38.486] > Waiting for server log... [23:26:38.518] > Waiting for server log... [23:26:38.551] > Waiting for server log... [23:26:38.583] > Waiting for server log... [23:26:38.615] > Waiting for server log... [23:26:38.647] > Waiting for server log... [23:26:38.679] > Waiting for server log... [23:26:38.711] > Waiting for server log... [23:26:38.744] > Waiting for server log... [23:26:38.776] > Waiting for server log... [23:26:38.808] > Waiting for server log... [23:26:38.840] > Waiting for server log... [23:26:38.872] > Waiting for server log... [23:26:38.905] > Waiting for server log... [23:26:38.936] > Waiting for server log... [23:26:38.969] > Waiting for server log... [23:26:39.001] > Waiting for server log... [23:26:39.033] > Waiting for server log... [23:26:39.066] > Waiting for server log... [23:26:39.098] > Waiting for server log... [23:26:39.130] > Waiting for server log... [23:26:39.163] > Waiting for server log... [23:26:39.195] > Waiting for server log... [23:26:39.227] > Waiting for server log... [23:26:39.259] > Waiting for server log... [23:26:39.291] > Waiting for server log... [23:26:39.323] > Waiting for server log... [23:26:39.356] > Waiting for server log... [23:26:39.387] > Waiting for server log... [23:26:39.420] > Waiting for server log... [23:26:39.452] > Waiting for server log... [23:26:39.484] > Waiting for server log... [23:26:39.517] > Waiting for server log... [23:26:39.549] > Waiting for server log... [23:26:39.581] > Waiting for server log... [23:26:39.613] > Waiting for server log... [23:26:39.646] > Waiting for server log... [23:26:39.678] > Waiting for server log... [23:26:39.712] > Waiting for server log... [23:26:39.742] > Waiting for server log... [23:26:39.775] > Waiting for server log... [23:26:39.806] > Waiting for server log... [23:26:39.838] > Waiting for server log... [23:26:39.871] > Waiting for server log... [23:26:39.903] > Waiting for server log... [23:26:39.935] > Waiting for server log... [23:26:39.968] > Waiting for server log... [23:26:40.000] > Waiting for server log... [23:26:40.032] > Waiting for server log... [23:26:40.064] > Waiting for server log... [23:26:40.096] > Waiting for server log... [23:26:40.129] > Waiting for server log... [23:26:40.161] > Waiting for server log... [23:26:40.199] > Waiting for server log... [23:26:40.226] > Waiting for server log... [23:26:40.257] > Waiting for server log... [23:26:40.290] > Waiting for server log... [23:26:40.323] > Waiting for server log... [23:26:40.356] > Waiting for server log... [23:26:40.387] > Waiting for server log... [23:26:40.419] > Waiting for server log... [23:26:40.451] > Waiting for server log... [23:26:40.532] > Waiting for server log... > Waiting for server log... [23:26:40.548] > Waiting for server log... [23:26:40.580] > Waiting for server log... [23:26:40.614] > Waiting for server log... [23:26:40.645] > Waiting for server log... [23:26:40.678] > Waiting for server log... [23:26:40.709] > Waiting for server log... [23:26:40.742] > Waiting for server log... [23:26:40.774] > Waiting for server log... [23:26:40.806] > Waiting for server log... [23:26:40.838] > Waiting for server log... [23:26:40.870] > Waiting for server log... [23:26:40.902] > Waiting for server log... [23:26:40.936] > Waiting for server log... [23:26:40.967] > Waiting for server log... [23:26:41.048] > Waiting for server log... > Waiting for server log... [23:26:41.064] > Waiting for server log... [23:26:41.096] > Waiting for server log... [23:26:41.128] > Waiting for server log... [23:26:41.160] > Waiting for server log... [23:26:41.197] > Waiting for server log... [23:26:41.224] > Waiting for server log... [23:26:41.256] > Waiting for server log... [23:26:41.288] > Waiting for server log... [23:26:41.322] > Waiting for server log... [23:26:41.353] > Waiting for server log... [23:26:41.385] > Waiting for server log... [23:26:41.417] > Waiting for server log... [23:26:41.498] > Waiting for server log... > Waiting for server log... [23:26:41.515] > Waiting for server log... [23:26:41.547] > Waiting for server log... [23:26:41.579] > Waiting for server log... [23:26:41.611] > Waiting for server log... [23:26:41.643] > Waiting for server log... [23:26:41.675] > Waiting for server log... [23:26:41.708] > Waiting for server log... [23:26:41.740] > Waiting for server log... [23:26:41.772] > Waiting for server log... [23:26:41.804] > Waiting for server log... [23:26:41.837] > Waiting for server log... [23:26:41.876] > Waiting for server log... [23:26:41.901] > Waiting for server log... [23:26:41.933] > Waiting for server log... [23:26:41.965] > Waiting for server log... [23:26:41.998] > Waiting for server log... [23:26:42.030] > Waiting for server log... [23:26:42.062] > Waiting for server log... [23:26:42.095] > Waiting for server log... [23:26:42.127] > Waiting for server log... [23:26:42.159] > Waiting for server log... [23:26:42.190] > Waiting for server log... [23:26:42.223] > Waiting for server log... [23:26:42.256] > Waiting for server log... [23:26:42.336] > Waiting for server log... > Waiting for server log... [23:26:42.353] > Waiting for server log... [23:26:42.385] > Waiting for server log... [23:26:42.418] > Waiting for server log... [23:26:42.449] > Waiting for server log... [23:26:42.529] > Waiting for server log... > Waiting for server log... [23:26:42.546] > Waiting for server log... [23:26:42.578] > Waiting for server log... [23:26:42.610] > Waiting for server log... [23:26:42.642] > Waiting for server log... [23:26:42.674] > Waiting for server log... [23:26:42.706] > Waiting for server log... [23:26:42.739] > Waiting for server log... [23:26:42.771] > Waiting for server log... [23:26:42.804] > Waiting for server log... [23:26:42.835] > Waiting for server log... [23:26:42.868] > Waiting for server log... [23:26:42.900] > Waiting for server log... [23:26:42.933] > Waiting for server log... [23:26:42.965] > Waiting for server log... [23:26:42.998] > Waiting for server log... [23:26:43.030] > Waiting for server log... [23:26:43.062] > Waiting for server log... [23:26:43.094] > Waiting for server log... [23:26:43.127] > Waiting for server log... [23:26:43.159] > Waiting for server log... [23:26:43.191] > Waiting for server log... [23:26:43.224] > Waiting for server log... [23:26:43.255] > Waiting for server log... [23:26:43.289] > Waiting for server log... [23:26:43.321] > Waiting for server log... [23:26:43.353] > Waiting for server log... [23:26:43.385] > Waiting for server log... [23:26:43.417] > Waiting for server log... [23:26:43.449] > Waiting for server log... [23:26:43.481] > Waiting for server log... [23:26:43.514] > Waiting for server log... [23:26:43.546] > Waiting for server log... [23:26:43.579] > Waiting for server log... [23:26:43.611] > Waiting for server log... [23:26:43.643] > Waiting for server log... [23:26:43.675] > Waiting for server log... [23:26:43.708] > Waiting for server log... [23:26:43.740] > Waiting for server log... [23:26:43.772] > Waiting for server log... [23:26:43.804] > Waiting for server log... [23:26:43.836] > Waiting for server log... [23:26:43.918] > Waiting for server log... > Waiting for server log... [23:26:43.934] > Waiting for server log... [23:26:43.966] > Waiting for server log... [23:26:43.997] > Waiting for server log... [23:26:44.030] > Waiting for server log... [23:26:44.062] > Waiting for server log... [23:26:44.094] > Waiting for server log... [23:26:44.126] > Waiting for server log... [23:26:44.159] > Waiting for server log... [23:26:44.193] > Waiting for server log... [23:26:44.223] > Waiting for server log... [23:26:44.255] > Waiting for server log... [23:26:44.287] > Waiting for server log... [23:26:44.368] > Waiting for server log... > Waiting for server log... [23:26:44.385] > Waiting for server log... [23:26:44.416] > Waiting for server log... [23:26:44.448] > Waiting for server log... [23:26:44.481] > Waiting for server log... [23:26:44.513] > Waiting for server log... [23:26:44.546] > Waiting for server log... [23:26:44.578] > Waiting for server log... [23:26:44.610] > Waiting for server log... [23:26:44.642] > Waiting for server log... [23:26:44.674] > Waiting for server log... [23:26:44.706] > Waiting for server log... [23:26:44.739] > Waiting for server log... [23:26:44.770] > Waiting for server log... [23:26:44.803] > Waiting for server log... [23:26:44.884] > Waiting for server log... > Waiting for server log... [23:26:44.899] > Waiting for server log... [23:26:44.931] > Waiting for server log... [23:26:44.963] > Waiting for server log... [23:26:44.996] > Waiting for server log... [23:26:45.028] > Waiting for server log... [23:26:45.060] > > * [23:26:45.076] > * Visual Studio Code Server > * > * By using the software, you agree to > * the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and > * the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement). > * > > Server did not start successfully. Full server log at /home/a1018048/.vscode-server/.8b3775030ed1a69b13e4f4c628c612102e30a681.log >>> [23:26:45.077] > node:internal/modules/cjs/loader:1075 > const err = new Error(message); > ^ > > Error: Cannot find module 'minimist' > Require stack: > - /home/a1018048/.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/out/server-main.js > at Module._resolveFilename (node:internal/modules/cjs/loader:1075:15) > at Module._load (node:internal/modules/cjs/loader:920:27) > at Module.require (node:internal/modules/cjs/loader:1141:19) > at require (node:internal/modules/cjs/helpers:110:18) > at start (/home/a1018048/.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/out/server-main.js:4:10642) > at Object. (/home/a1018048/.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/out/server-main.js:15:28) > at Module._compile (node:internal/modules/cjs/loader:1254:14) > at Module._extensions..js (node:internal/modules/cjs/loader:1308:10) > at Module.load (node:internal/modules/cjs/loader:1117:32) > at Module._load (node:internal/modules/cjs/loader:958:12) { > code: 'MODULE_NOT_FOUND', > requireStack: [ > '/home/a1018048/.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/out/server-main.js' > ] > } > > Node.js v18.15.0 > <<< End of server log > cf1ea10c7283: start > exitCode==32== > listeningOn==== > osReleaseId==rhel== > arch==x86_64== > vscodeArch==x64== > bitness==64== > tmpDir==/run/user/1018048== > platform==linux== > unpackResult==== > didLocalDownload==0== > downloadTime==== > installTime==== > extInstallTime==== [23:26:45.093] > serverStartTime==14551== > cf1ea10c7283: end [23:26:45.093] Received install output: exitCode==32== listeningOn==== osReleaseId==rhel== arch==x86_64== vscodeArch==x64== bitness==64== tmpDir==/run/user/1018048== platform==linux== unpackResult==== didLocalDownload==0== downloadTime==== installTime==== extInstallTime====serverStartTime==14551== [23:26:45.093] Terminating local server [23:26:45.096] Resolver error: Error: The VS Code Server failed to start at g.ServerInstallError (/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:642637) at d (/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:636024) at t.handleInstallOutput (/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:638294) at e (/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:700300) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:722169 at async t.withShowDetailsEvent (/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:725828) at async /home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:697005 at async C (/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:695056) at async t.resolveWithLocalServer (/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:696545) at async I (/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:719392) at async t.resolve (/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:723199) at async /home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/extension.js:2:907003 [23:26:45.100] ------ [23:26:45.103] Local server exit: 15 [23:26:45.417] Opening exec server for ssh-remote+phoenix [23:26:45.429] Initizing new exec server for ssh-remote+phoenix [23:26:45.450] Acquiring local install lock: /tmp/vscode-remote-ssh-74cd3cec-install.lock [23:26:45.456] Looking for existing server data file at /home/steviep/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-74cd3cec-8b3775030ed1a69b13e4f4c628c612102e30a681-0.107.1-es/data.json [23:26:45.460] Using commit id "8b3775030ed1a69b13e4f4c628c612102e30a681" and quality "stable" for server [23:26:45.462] Install and start server if needed [23:26:45.464] askpass server listening on /run/user/1000/vscode-ssh-askpass-179aba1d30c02e7a86f77e1c55070193b12059d8.sock [23:26:45.464] Spawning local server with {"serverId":2,"ipcHandlePath":"/run/user/1000/vscode-ssh-askpass-a55fe519608b24653ac2501fb433689ce255cff5.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","44835","-o","ConnectTimeout=15","phoenix"],"serverDataFolderName":".vscode-server","dataFilePath":"/home/steviep/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-74cd3cec-8b3775030ed1a69b13e4f4c628c612102e30a681-0.107.1-es/data.json"} [23:26:45.464] Local server env: {"SSH_AUTH_SOCK":"/run/user/1000/keyring/ssh","SHELL":"/bin/bash","DISPLAY":":0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/usr/share/code/code","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/home/steviep/.vscode/extensions/ms-vscode-remote.remote-ssh-0.107.1/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/run/user/1000/vscode-ssh-askpass-179aba1d30c02e7a86f77e1c55070193b12059d8.sock"} [23:26:45.467] Spawned 6235 [23:26:45.563] > local-server-2> Running ssh connection command: "-v -T -D 44835 -o ConnectTimeout=15 phoenix" [23:26:45.566] > local-server-2> Spawned ssh, pid=6243 [23:26:45.568] stderr> OpenSSH_8.2p1 Ubuntu-4ubuntu0.11, OpenSSL 1.1.1f 31 Mar 2020 [23:26:50.691] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:R9IY5h4bjTgb4djJQ6G2qQ92s3JQEvVkAfNX2Rc9x2A [23:26:50.846] stderr> No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000) [23:26:50.847] stderr> [23:26:50.847] stderr> [23:26:50.899] stderr> No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000) [23:26:50.899] stderr> [23:26:50.899] stderr> [23:26:50.993] Got askpass request: {"request":"a1018048@p2-log-1.hpc.adelaide.edu.au's password:"} [23:26:50.993] Showing password prompt [23:26:50.994] Listening for interwindow password on /run/user/1000/vscode-ssh-askpass-6461fb0bf0d13319ad46516c2f46374d7ad12bec.sock [23:26:50.994] Writing password prompt to globalState ```

Anything else?

I've been connecting to 3 remote servers very successfully for a while, but have noticed that recently, I'm required to enter my password twice on each of them. I don't know if this is related at all

microsoft / vscode-remote-release