Remote development: SSH connection fails, manual connection inconsistent

[WiseHacks]

Check for existing issues

• [X] Completed

Describe the bug / provide steps to reproduce it

Context: there are two servers - one is jumphost server and other is remote machine, i want to connect: mac -> remote. we also use VPN service. While trying different ways to connect via ssh, got these bugs:

1. ssh -J jumphost1 devserver - Fails, this even fails if ran from terminal because it can not recognize proper authentication.
2. ssh -J jumphost2 devserver - Fails, same as (1) but difference is that it works only with vpn service enabled while jumphost1 uses id_rsa.
3. (1) got connected once, might be because of ISP but i can not remember how, putting that aside - when opening terminal in remote connection, noticed that it was opening and getting closed continuously (this might be an issue), but this got connected as well in few tries after pauses (can't think of some reason)

4. What actually works in vs code remote extension and terminal is:

   • consider following ssh config:

     
     Host jumphost2
     IdentitiesOnly yes
     UserKnownHostsFile=/dev/null
     StrictHostKeyChecking no
     LogLevel quiet
     HostName jumphost2.com
     User me
     IdentityFile "some/where/to/the/key.pem"

   Host devserver HostName ip User me ProxyCommand ssh -o "ForwardAgent yes" -o "IdentitiesOnly yes" -o UserKnownHostsFile=/dev/null -o "StrictHostKeyChecking no" -o "LogLevel quiet" jumphost2 "custom-ssh-wrapper && nc %h %p"

   
   `ssh devserver` - Fails, with error - 

   nc: invalid option -- 'c' usage: nc [-46CDdFhklNnrStUuvZz] [-I length] [-i interval] [-M ttl] [-m minttl] [-O length] [-P proxy_username] [-p source_port] [-q seconds] [-s source] [-T keyword] [-V rtable] [-W recvlimit] [-w timeout] [-X proxy_protocol] [-x proxy_address[:port]] [destination] [port] kex_exchange_identification: Connection closed by remote host

   **Update 1** 
   when tried the command in verbose mode: `ssh devserver -vvv`, got this in task output

   OpenSSH_9.4p1, LibreSSL 3.3.6 debug1: Reading configuration data /Users/username/.ssh/config debug1: /Users/username/.ssh/config line 24: Applying options for devserver debug1: /Users/username/.ssh/config line 38: Skipping Host block because of negated match for devserver debug1: /Users/username/.ssh/config line 46: include ~/.ssh/bnn.config matched no files debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/ matched no files debug1: /etc/ssh/ssh_config line 54: Applying options for debug2: resolve_canonicalize: hostname 192.xx.xx.xx is address debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/username/.ssh/known_hosts' debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling debug1: Executing proxy command: exec ssh -o "ForwardAgent yes" -o "IdentitiesOnly yes" -o UserKnownHostsFile=/dev/null -o "StrictHostKeyChecking no" -o "LogLevel quiet" jumphost2 "custom-ssh-wrapper && nc 192.xx.xx.xx 22" debug1: identity file /Users/username/.ssh/id_rsa type -1 debug1: identity file /Users/username/.ssh/id_rsa-cert type -1 debug1: identity file /Users/username/.ssh/id_ecdsa type -1 debug1: identity file /Users/username/.ssh/id_ecdsa-cert type -1 debug1: identity file /Users/username/.ssh/id_ecdsa_sk type -1 debug1: identity file /Users/username/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /Users/username/.ssh/id_ed25519 type -1 debug1: identity file /Users/username/.ssh/id_ed25519-cert type -1 debug1: identity file /Users/username/.ssh/id_ed25519_sk type -1 debug1: identity file /Users/username/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /Users/username/.ssh/id_xmss type -1 debug1: identity file /Users/username/.ssh/id_xmss-cert type -1 debug1: identity file /Users/username/.ssh/id_dsa type -1 debug1: identity file /Users/username/.ssh/id_dsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_9.4 nc: invalid option -- 'c' usage: nc [-46CDdFhklNnrStUuvZz] [-I length] [-i interval] [-M ttl] [-m minttl] [-O length] [-P proxy_username] [-p source_port] [-q seconds] [-s source] [-T keyword] [-V rtable] [-W recvlimit] [-w timeout] [-X proxy_protocol] [-x proxy_address[:port]] [destination] [port] kex_exchange_identification: Connection closed by remote host Connection closed by UNKNOWN port 65535

⏵ Task Install zed over ssh finished with non-zero error code: 255 ⏵ Command: 'ssh devserver -vvv'

I am not entirely sure if this is a bug or not, but it works in vs code so should work here too (please make it work :D)

If only 4th option can work, then everything is sorted, terminal would work in remote connection too i guess.
coming to manual connection, there is no problem in this but terminal does not work which is known however in between of few connection, the headless instance was getting killed when opening a project. also got this error when tried to remove the dev-server from the list: 
<img width="371" alt="image" src="https://github.com/zed-industries/zed/assets/152850436/1b4a5e80-410c-4295-971d-497a02f96b83">
after some pause, it was deleted fine.

on a side note, it would be fine if i can have reconnect option on side of dev-server name, because when the server is disconnected abruptly and editor can not know, the "update" option does not appear in manual connection and i have to change the state of input command box (i mean delete one character, add again).

### Environment

Zed: v0.137.2 (Zed Preview)
OS: macOS 14.1.0
Memory: 8 GiB
Architecture: aarch64

### If applicable, add mockups / screenshots to help explain present your vision of the feature

_No response_

### If applicable, attach your `~/Library/Logs/Zed/Zed.log` file to this issue.

_No response_

[WiseHacks]

Hey folks, so i was playing with the feature by tweaking my config and reading through verbose - In the 4th option where the ProxyCommand option is used, turns out that there is some other issue. basically when we run some command, in my case it was custom-ssh-wrapper nc %h %p but consider something like date && echo hey, this is the task output then ->

debug1: Local version string SSH-2.0-OpenSSH_9.4
debug1: kex_exchange_identification: banner line 0: Thu May 30 15:15:10 IST 2024
debug1: kex_exchange_identification: banner line 1: hey sh -c cd;  sh -x -c '~/.local/bin/zed -v >/dev/stderr || (curl -sSL https://zed.dev/install.sh || wget -qO- https://zed.dev/install.sh) | bash && ~/.local/bin/zed --dev-server-token 197.B54QecTR5-CHv6xlUYn3lDhdT_hX7Y35128ACTe4'
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

notice the -c there - it is the problem, actually if i use ; at the end of my command (i.e custom-ssh-wrapper nc %h %p;), it works just fine. So this was the culprit :/

(this -c, may be adding ; or && prefix in zed should fix it)

In conclusion,

1. ssh connection works just fine so can ignore that, -J was not working in terminal too so that is not really a bug. however new terminal connections are very slow compared to other remote dev editors but that is another story of improvements.
2. Manual connection issues still hold (disconnection, UI not redirecting to main screen when remote path is not reachable etc)

Thanks! I can finally use it for my work, only waiting for notebook support here to fully shift on zed. Keep up the good work :)

[ConradIrwin]

Thanks! To reframe to make sure I understand:

• SSH-based connecting is now fixed by adding a ; in your ssh config?
• You still see the connection disconnecting frequently?

For the second issue, could you please upload the zed log (~/.local/share/zed/logs/Zed.log on your remote instance). If you don't feel comfortable uploading it you can also email it to conrad@zed.dev

It may also be worth checking the version of zed you're running on the remote (zed -v) as you started using this feature pretty early. It should be v0.138 or greater (you can upgrade with curl https://zed.dev/install.sh | bash if needed).

[WiseHacks]

Hi, sorry couldn't get back earlier.

Yes adding ';' fixed the connection. I've not used the manual connection much since i am used to work with integrated terminal but have been trying to reproduce it, but it seems better in terms of connection now, Though there are some other problems i am facing; not specific to manual connection rather remote development. Like code formatter (external) not working, project panel disappearing randomly after some idle time (adding screenshot

), scroll by touch-pad does not work in terminal (less pager) (well its a general issue not related with remote-dev)

once again, apologies for delayed response!

[notpeter]

@WiseHacks I'm going to close this since it fixed the connection issue.

If you're still having issues with zed features (code formatting, etc) being broken could you please create a new issue describing that?

Thanks.