"Connecting with SSH timed out" when Fish/tcsh/csh shell is used

[tals]

workaround: in preferences, set remote.SSH.useLocalServer to true and remote.SSH.remotePlatform like so:

"remote.SSH.remotePlatform": {
        <hostname1>: "linux",
        <hostname2>: "linux"
    },

workaround: Downgrading this extension to 0.49.0 fixes the issue (although it appears to break the "recent connections" UI). Extensions menu -> Right-click on "Remote - SSH" -> "Install Another Version" -> 0.49.0

This is a regression on #33, started after I upgraded to VScode 1.43.0/extension 0.50.0.

• VSCode Version: 1.43.0
• Local OS Version: macOS 10.14.6
• Remote OS Version: Ubuntu 18.04.3 LTS
• Remote Extension/Connection Type: SSH

Steps to Reproduce:

1. Set bash to default shell
2. Connect with VSCode remote. Works as expected.
3. Set default remote shell as Fish
4. Connecting with VSCode times out.

This was observed on two different servers

Does this issue occur when you try this locally?: N/A Does this issue occur when you try this locally and all extensions are disabled?: N/A

[11:40:01.637] Log Level: 2
[11:40:01.639] remote-ssh@0.50.0
[11:40:01.639] darwin x64
[11:40:01.640] SSH Resolver called for "ssh-remote+anton", attempt 1
[11:40:01.640] SSH Resolver called for host: anton
[11:40:01.640] Setting up SSH remote "anton"
[11:40:01.643] Acquiring local install lock: /var/folders/y0/x378plh96hndqvy04th4v7880000gn/T/vscode-remote-ssh-anton-install.lock
[11:40:01.654] Looking for existing server data file at /Users/tal/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-anton-78a4c91400152c0f27ba4d363eb56d2835f9903a/data.json
[11:40:01.655] Using commit id "78a4c91400152c0f27ba4d363eb56d2835f9903a" and quality "stable" for server
[11:40:01.656] Install and start server if needed
[11:40:01.659] Checking ssh with "ssh -V"
[11:40:01.673] > OpenSSH_7.9p1, LibreSSL 2.7.3
[11:40:01.676] askpass server listening on /var/folders/y0/x378plh96hndqvy04th4v7880000gn/T/vscode-ssh-askpass-298a99f5c46843bde64ed3c630b85b9c9c6619df.sock
[11:40:01.677] Spawning local server with {"ipcHandlePath":"/var/folders/y0/x378plh96hndqvy04th4v7880000gn/T/vscode-ssh-askpass-42adc0358caea0dce208ba088ea9f67fc0396523.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","56862","-o","ConnectTimeout=30","-o","RemoteCommand=none","anton"],"dataFilePath":"/Users/tal/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-anton-78a4c91400152c0f27ba4d363eb56d2835f9903a/data.json"}
[11:40:01.677] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/tal/.vscode/extensions/ms-vscode-remote.remote-ssh-0.50.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)","VSCODE_SSH_ASKPASS_MAIN":"/Users/tal/.vscode/extensions/ms-vscode-remote.remote-ssh-0.50.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/y0/x378plh96hndqvy04th4v7880000gn/T/vscode-ssh-askpass-298a99f5c46843bde64ed3c630b85b9c9c6619df.sock"}
[11:40:01.679] Spawned 93505
[11:40:01.793] > local-server> Spawned ssh: 93506
[11:40:01.803] stderr> OpenSSH_7.9p1, LibreSSL 2.7.3
[11:40:03.569] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:bFQr5QDzdBZHWOYB8GrdsncBvTdS47Ebd7x5ZKg/GKA
[11:40:04.998] stderr> Authenticated to <hidden> ([<hidden>]:22).
[11:40:05.920] > Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 5.0.0-37-generic x86_64)
> 
>  * Documentation:  https://help.ubuntu.com
>  * Management:     https://landscape.canonical.com
>  * Support:        https://ubuntu.com/advantage
> 
>  * Latest Kubernetes 1.18 beta is now available for your laptop, NUC, cloud
>    instance or Raspberry Pi, with automatic updates to the final GA release.
> 
>      sudo snap install microk8s --channel=1.18/beta --classic
> 
>  * Multipass 1.1 adds proxy support for developers behind enterprise
>    firewalls. Rapid prototyping for cloud operations just got easier.
> 
>      https://multipass.run/
> 
>  * Canonical Livepatch is available for installation.
>    - Reduce system reboots and improve kernel security. Activate at:
>      https://ubuntu.com/livepatch
> 
> 174 packages can be updated.
> 0 updates are security updates.
> 
> Your Hardware Enablement Stack (HWE) is supported until April 2023.
> *** System restart required ***
[11:40:33.680] Terminating local server
[11:40:33.681] Resolver error: Connecting with SSH timed out
[11:40:33.685] ------

[11:40:33.687] Local server exit: 15

[cameronbraid]

I can confirm the same issue

[graffic]

I can confirm, using fish shell, hell broke loose.

At firsh ssh-remote was complaining about requiring a non-windows ssh to work. I throught that was the issue but after installing git with ssh, the problem did not go away.

I just downgraded to 0.49, as suggested by @tals, and it works again.

[zmx264]

Same issue here, does not work anymore on fish shell

[qhhonx]

I can confirm, using fish shell, hell broke loose.

At firsh ssh-remote was complaining about requiring a non-windows ssh to work. I throught that was the issue but after installing git with ssh, the problem did not go away.

I just downgraded to 0.49, as suggested by @tals, and it works again.

How can it downgrade to 0.49 ?

[graffic]

How can it downgrade to 0.49 ?

@qhhonx In the extensions menu, search for the remote-ssh extension and in the bottom right of the ssh-remote item, you will see a cog icon. Press it and you will see a menu with an option to install other versions.

[qhhonx]

How can it downgrade to 0.49 ?

@qhhonx In the extensions menu, search for the remote-ssh extension and in the bottom right of the ssh-remote item, you will see a cog icon. Press it and you will see a menu with an option to install other versions.

Big thanks for you, it works again as expected.

[roblourens]

As a workaround for 0.50, you could try setting "remote.SSH.useLocalServer": false, and let me know whether that helps.

[zmx264]

As a workaround for 0.50, you could try setting "remote.SSH.useLocalServer": false, and let me know whether that helps.

yes, setting useLocalServer to false solves the issue.

[tals]

As a workaround for 0.50, you could try setting "remote.SSH.useLocalServer": false, and let me know whether that helps.

Can confirm this works

[cameronbraid]

remote.SSH.useLocalServer=false fixes it for me too

[michaelaye]

same problem here, was using fish for months without problems, suddenly failed (macOS 13.0.4), workaround also works here.

[roblourens]

Sorry for the trouble everyone. I can't figure out what fish is doing here to break this. I just spawn an ssh process and pipe a script into stdin, but any command sent that way is simply ignored when fish is used.

[graffic]

So I tried the other fix: remote.SSH.useLocalServer=false and it fixes it for me too. I guess there is already an issue about local server enabled automatically so perhaps this issue is duplicated.

[tals]

Sorry for the trouble everyone. I can't figure out what fish is doing here to break this. I just spawn an ssh process and pipe a script into stdin, but any command sent that way is simply ignored when fish is used.

@roblourens huh. judging by the other bugs here, it seems to happen with other shells like tcsh, so maybe its not fish specific.

After some experiments, I can see that fish wont process input being written to ssh if theres no terminal attached.

const childProcess = require('child_process')
const process = require('process')
const myServer = 'mg'
const p = childProcess.spawn(
    'ssh',
    // this wont print "hi there"
    // ["-v","-T", "-o","ConnectTimeout=30",myServer],
    // this will. note the "-tt" to attach a pseudo tty
    ["-v","-tt", "-o","ConnectTimeout=30",myServer],
    { stdio: 'pipe'}
)
p.stdin.write('echo "hi there!"\n')

p.stdout.on('data', e => {
    process.stdout.write(e)
})

p.stderr.on('data', e=> {
    process.stderr.write(e)
})

The reason it works without the local server is because the input stream terminates. You can see it by doing:

# this will print "hi"
$ echo "echo hi" | ssh -T mg

# this wont print anything
$ yes "echo hi" | ssh -T mg

# This print "hi"s forever
$ yes "echo hi" | ssh -T mg bash

# So will this (plus some shell prompts)
$ yes "echo hi" | ssh -tt mg

[tals]

Sorry for the trouble everyone. I can't figure out what fish is doing here to break this. I just spawn an ssh process and pipe a script into stdin, but any command sent that way is simply ignored when fish is used.

Hey @roblourens, I think I found a workaround that allows the server to continue running. In getSshConnectionArgs(), simply append "bash" to the ssh command, which skips fish. I patched the extension locally and it seems to work. The integrated terminal still works.

(I couldn't find the source code on github. Would've opened a PR otherwise)

[roblourens]

Yes, that does work. However, I want to be able to attach to windows, mac, or linux remotes without knowing what platform they are ahead of time, so I can't set bash as an initial command.

Thanks for your investigation, I have a nearly identical script 😁 except that I had tried -t and -ttt (thinking of -vvv) neither of which work, and -tt actually does. It's very awkward that it causes the input and a prompt to be echoed. And I'm nervous about introducing other issues by trying it in general. It might be safer to just say that people using fish have to set a setting (either to disable local server mode or to do opt-into the -tt flag). I get a mess of escape characters, and I will have to strip tab characters from the install script because they trigger actual tab completion. So I'm not looking forward to the weird issues that might come up if I do this for every connection. But this is a good hint about a possible path forward, thanks!

[michaelaye]

is this related? https://github.com/fish-shell/fish-shell/issues/1439

[tals]

@roblourens ah I see.

You could run a probe for the platform in a preflight call. This is stable, because stdin would get closed as soon as uname -rsv gets dispatched by probeServerPlatform(), similarly to how it is when useLocalServer is off.

Once you know you're not on windows, you can go ahead with the bash bit (the mainScriptWrapper bit is simply a bash script, and its always used if you're not running windows).

The main issue I see with this is that the preflight would require an extra askpass for people without ssh keys, which is annoying to the user, but I think the useLocalServer=off bit has the same problem.

In theory once you've bootstrapped the remote machine, you can have a remote node agent that binds on a local port on the remote machine. localServer then runs it together with SSH port forwarding, and you use this ssh tunnel to send commands to the remote server, which runs them.

[tals]

is this related? fish-shell/fish-shell#1439

ha maybe. Good find!

[roblourens]

The main issue I see with this is that the preflight would require an extra askpass for people without ssh keys, which is annoying to the user, but I think the useLocalServer=off bit has the same problem.

Many things would be simpler if I could run multiple ssh commands, but I have managed to keep it to a single ssh connection at least per window, and I don't want to backtrack on that since many people use passwords, 2fa, etc which are very annoying to enter multiple times.

In theory once you've bootstrapped the remote machine, you can have a remote node agent that binds on a local port on the remote machine. localServer then runs it together with SSH port forwarding, and you use this ssh tunnel to send commands to the remote server, which runs them.

Well, this is exactly what we are doing, but for every connection we still have to run a script to determine whether the remote server is already running, or needs to be updated, or which port it is listening on, etc.

[tals]

Many things would be simpler if I could run multiple ssh commands, but I have managed to keep it to a single ssh connection at least per window, and I don't want to backtrack on that since many people use passwords, 2fa, etc which are very annoying to enter multiple times.

@roblourens Makes sense. That's tricky! My understanding is that this was the case before useLocalServer was enabled by default in 0.50 -- you had to do at least two ssh connections: the preflight one and then the bootstrap one, so backtracking from that would lose some of the advantages of localServer.

How do you feel about caching the platform information, so its amortized?

I wonder if you could reliably infer if you're dealing with windows vs linux based on stuff coming back when you're doing the ssh handshake. For example if I run ssh -T -v <server>, I get some server platform information back:

debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002

Here I can tell this is ubuntu linux before any command has been issued.

I don't have a windows machine available, but maybe it returns similar information so you could fingerprint it (and might be easier to fingerprint than linux haha)

[bobthecow]

Same issue here (macOS Catalina locally, some flavor of Ubuntu with a reasonably modern Fish remote). Downgrading fixed it, but setting "remote.SSH.useLocalServer" to false did not.

In fact, that was even worse. Somehow instead of just hanging, disabling it resulted in a really cool crash-loopy thing where it kept trying to connect and failing as fast as it could start, popping terminal windows open and closed.

[roblourens]

@tals, without useLocalServer we still do one connection, but we have to know the platform ahead of time, assuming Linux unless the windowsRemotes setting contains the current host.

Connecting multiple times the first time and caching that info is a possibility but I really want the first experience to be good. Also considered just showing a popup for the user to select the platform, then keeping that info in a setting.

@bobthecow could you open a new issue and share which remote.SSH settings you have set, and the log from the Remote-SSH channel?

[chiraag]

@roblourens I think asking for prompt on first use and then caching the information is a good compromise. As you are probably aware from the various bug-reports the current extension also breaks on (t)csh shells which are fairly common on RHEL/CentOS machines.

[roblourens]

Yeah, that issue I have not been able to reproduce however.

[chiraag]

@roblourens The failure mode is almost exactly the same as with fish. Here is a similar experiment to @tals with tcsh being the login shell on the remote machine.

$ cat > test_tcsh.sh <<EOF
bash
function test_fn {
        echo "> in test_fn"
}
test_fn
EOF

$ cat test_tcsh.sh | ssh -T remote_host
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
> in test_fn
function: Command not found.
}: Command not found.
test_fn: Command not found.

$ cat test_tcsh.sh | ssh -T remote_host bash
> in test_fn

[roblourens]

Hm, I can repro it now, I don't know what I was doing wrong before. I agree that it's probably basically the same thing.

[david50407]

Set these config works for me:

"remote.SSH.showLoginTerminal": true,
"remote.SSH.useLocalServer": false,

VScode 1.43.2 ssh -V: OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5

remote fish -v: fish, version 2.7.1

[benny-edlund]

I'm see this issue as well when changing my local machine from a surface laptop to a Mac.

Still connecting to the same remote host that uses tcsh.

Default shell Mac is also tcsh.

Windows10 works fine however Mac is authenticating but breaks when seemingly sending bash commands to the remote tcsh shell

useLocalServer : false does help but you get stuck with this install terminal that seems to own the connection to the remote host

Also user experience is quite poor when needing to reconnect as you seem to get another install terminal

Is it possible the issue lies with the communication with the local server rather then the remote?

[vibhoothi]

Yes I did also face same issue when fish the default shell in the remote shell(arm) doing the temporary workaround did work fine

[fuzihaofzh]

Setting "remote.SSH.useLocalServer": false," in settings.json works for me. You can open this file by File -> Preferences -> Settings -> Extensions -> Scroll down and find "Edit in settings.json"

[karan173]

Any updates on a fix? Would be nice to have the functionality of remote.SSH.useLocalServer: true while using fish. Are there any local workarounds which you could suggest to make the combination work?

[fengyuentau]

Even if I uninstalled fish shell, this issue still happens. And I have tried setting the localServer to false, but it did not help. What I did to solve the issue at last is downgrading the Remote-SSH to version 0.49 following @graffic 's advice. If you find the button Install other version is gray and not clickable, try uninstall and install, then install other version.

[ryan-allen]

The previous suggestions re: the config settings changes did not work for me. Changing my shell on the remote from fish to bash did work.

[jmcphers]

I also had this problem with zsh (and it was also resolved by changing to bash).

[takenaga]

I also had issue with csh, and 0.49.0 helped to solve.

Below is log from 0.49.0 and it expects bash on remote server, which @roblourens wants to avoid.

[22:46:27.244] Running script with connection command: ssh -T -D 54272 -o ConnectTimeout=15 <remote-host> bash

How about adding 'remote-shell' config parameter, and if it is specified, spawn that shell first and do the rest?

[JacobHenner]

Also experiencing this (with fish).

Are there plans to address this? The issue has been open since March, and the suggested workaround is limiting.

I am not one to complain about bugfix timelines in open source projects. However, this isn't an open source project, so even though I'd love to volunteer and submit a PR to fix the issue, I cannot.

[michaelaye]

@JacobHenner agree. I'm currently forced to default to bash on my remote, both ssh config and vsode workarounds do not work for me (Linux both client and host).

[MacJim]

Here's my (temporary) solution:

1. Use bash as your remote's default shell: chsh -s /bin/bash
2. At the end of your remote's ~/.bashrc file, add:

# Execute `fish` in interactive mode
if [ -z "$PS1" ]; then
        echo 'This shell is not interactive. Will keep using bash to make VS Code Remote work.'
else
        echo 'This shell is interactive. Will switch to fish.'
        exec fish
fi

The reasoning behind this: PS1 is unset in non-interactive shells (e.g. VSCode remote).

[tals]

Here's my (temporary) solution:

what a great trick @MacJim !

[michaelaye]

Interestingy, @MacJim 's trick doesn't work for me!

This is the log of vscode's Remote SSH trial, and I get the time-out error again. :(

[10:37:00.149] Log Level: 2
[10:37:00.150] remote-ssh@0.55.0
[10:37:00.150] linux x64
[10:37:00.151] SSH Resolver called for "ssh-remote+luna4", attempt 1
[10:37:00.151] SSH Resolver called for host: luna4
[10:37:00.151] Setting up SSH remote "luna4"
[10:37:00.153] Acquiring local install lock: /tmp/vscode-remote-ssh-luna4-install.lock
[10:37:01.653] Starting to look for password prompt from another window
[10:37:12.172] Looking for existing server data file at /home/maye/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-luna4-2af051012b66169dde0c4dfae3f5ef48f787ff69-0.55.0/data.json
[10:37:12.173] Using commit id "2af051012b66169dde0c4dfae3f5ef48f787ff69" and quality "stable" for server
[10:37:12.174] Install and start server if needed
[10:37:12.177] Checking ssh with "ssh -V"
[10:37:12.185] > OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020

[10:37:12.188] askpass server listening on /run/user/1000/snap.code/vscode-ssh-askpass-918e431a476f0b114408fb8ce6cdf3d59fa15414.sock
[10:37:12.188] Spawning local server with {"ipcHandlePath":"/run/user/1000/snap.code/vscode-ssh-askpass-ac1a26ae380bb9f82c98a6e502a76c546b3fdf77.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","44723","-o","ConnectTimeout=15","luna4"],"dataFilePath":"/home/maye/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-luna4-2af051012b66169dde0c4dfae3f5ef48f787ff69-0.55.0/data.json"}
[10:37:12.188] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/home/maye/.vscode/extensions/ms-vscode-remote.remote-ssh-0.55.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/snap/code/46/usr/share/code/code","VSCODE_SSH_ASKPASS_MAIN":"/home/maye/.vscode/extensions/ms-vscode-remote.remote-ssh-0.55.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/run/user/1000/snap.code/vscode-ssh-askpass-918e431a476f0b114408fb8ce6cdf3d59fa15414.sock"}
[10:37:12.191] Spawned 7179
[10:37:12.246] > local-server> Spawned ssh: 7187
[10:37:12.248] stderr> OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
[10:37:12.863] > This shell is interactive. Will switch to fish.

[MacJim]

Interestingy, @MacJim 's trick doesn't work for me!

Hmm. @michaelaye Did you modifyPS1 in .bashrc?

Maybe let's use the - special parameter instead. It contains i when the shell is interactive.

if [[ "$-" == *"i"* ]]; then
        exec fish
fi

[michaelaye]

Interestingy, @MacJim 's trick doesn't work for me!

Hmm. @michaelaye Did you modifyPS1 in .bashrc?

I think so?

()[maye@luna4 ~]$ echo $PS1
($(vcprompt -f %b%m))\[\033]0;\u:${NEW_PWD}\007\]\[\033[1;30m\][\[\033[0;37m\]\u\[\033[0;30m\]@\[\033[0;37m\]\h \[\033[1;34m\]${NEW_PWD}\[\033[1;30m\]]\[\033[0;37m\]\$ \[\033[0m\]

Maybe let's use the - special parameter instead. It contains i when the shell is interactive.

if [[ "$-" == *"i"* ]]; then
        exec fish
fi

Using that, I don't get fish when I log in via ssh. :(

[mdelaney]

I found a way to get this working without changing the shell for the remote user. Simply enable the setting Remote Server Listen On Socket.

NOTE: I've only tried this with fish shell, my remote machine is Ubuntu.

Of course, this assumes you are able/willing to set AllowStreamLocalForwarding on the remote SSH server.

[yatamlite]

My default shell is csh as well, and using version 0.49.0 was the only solution that worked for me since I couldn't change my shell using chsh or usermod.

Thanks a ton!

[dovahcrow]

Thank you @mdelaney ! This solution works for me for ubuntu 20.04 with fish.

[asetty]

had to change my default shell back to bash on the remote shell for now to unblock myself after getting excited about my new fish env. would be great to get this fixed.

Using version 0.49.0 as some others did also didn't work for me on an ARM machine.

[AlekSi]

workaround: in preferences, set remote.SSH.useLocalServer to false

That workaround worked for me like a charm

[tatsy]

My temporal solution:

• Use Byobu (I used tmux backend, but screen backend should be OK as well)
• Set Byobu's default shell as fish (configure $HOME/.byobu/.tmux.conf)
• Set sytem's default shell as zsh

Then, I can use vscode SSH through zsh, and can also use fish through Byobu when I want to do something with CLI.

[gioipv]

u can set the ssh time out is more than current time. It work for me. (I changed from 15 to 30)