SSH causes intermittent issues

[taman0753]

Discord username (optional)

No response

Describe the bug

When i do SSH into some sever this happens frequently .

channel 21: open failed: connect failed: open failed channel 21: open failed: connect failed: open failed channel 23: open failed: connect failed: open failed channel 21: open failed: connect failed: open failed channel 23: open failed: connect failed: open failed channel 25: open failed: connect failed: open failed channel 21: open failed: connect failed: open failed channel 23: open failed: connect failed: open failed channel 25: open failed: connect failed: open failed channel 21: open failed: connect failed: open failed channel 23: open failed: connect failed: open failed channel 25: open failed: connect failed: open failed channel 21: open failed: connect failed: open failed channel 21: open failed: connect failed: open failed channel 23: open failed: connect failed: open failed

To Reproduce

After doing ssh to machine.

Expected behaviour

No response

Screenshots

No response

Operating System

MacOS

OS Version

12.6

Shell Version

version 3.2.57

Warp Version

v0.2022.10.11.08.13.stable_01

Additional context

No response

Does this block you from using Warp daily?

Yes

Warp Internal (ignore): linear-label:b8107fdf-ba31-488d-b103-d271c89cac3e

No response

[orionarcher]

I came to report the same issue.

[taman0753]

@szgupta Can you check this

[szgupta]

Hey @taman0753, is this happening only in the latest release?

[taman0753]

@szgupta This was a intermittent issue is previous versions as well , but after update its very frequent which causes me to not use Warp terminal !!

[taman0753]

I have used the troubleshooting step given here https://docs.warp.dev/features/ssh#troubleshooting

[taman0753]

@szgupta @alokedesai Any updates

[alokedesai]

@taman0753 Just to confirm, you aren't seeing this in other terminals? Does this still repro in Warp if you disable the "Warp SSH Wrapper" setting?

[taman0753]

@alokedesai Other terminals are working as expected , after disabling "Warp SSH Wrapper" i am not able to utilise warp feature like suggestions , history etc.

[alokedesai]

@taman0753 Yeah, just wanted to make sure it was something specific to our SSH wrapper implementation :) Are you able to reproduce this for all the boxes you SSH into or is it only for specific ones?

[taman0753]

@alokedesai Its almost for all the boxes. after disabling "Warp SSH Wrapper" the issue is not seen as of now.

[taman0753]

@alokedesai But after disabling SSH wrapper functionalities are not working

[taman0753]

@alokedesai @szgupta Any updates here , please check its very frequent.

[alokedesai]

@taman0753 We're actively looking into this. Could you send us a video of the issue occurring so we can better understand when and how the error manifests?

[alokedesai]

@taman0753 Could you also run sudo sshd -T | grep -Ei 'TCPKeepAlive|AllowTCPForwarding|PermitOpen' on the remote boxes where this reproduces and send the output?

[taman0753]

@alokedesai Please find the output of the above command.

[taman0753]

@alokedesai I have shared the video over email (aloke@warp.dev) to you. Please acknowledge.

@taman0753 We're actively looking into this. Could you send us a video of the issue occurring so we can better understand when and how the error manifests?

[taman0753]

@alokedesai Let me know if any other thing needed to dig deeper to the issue.

[alokedesai]

@taman0753 Are you able to reproduce this if you turn off syntax highlighting and error underlining?

[orionarcher]

I disabled syntax highlighting and error underlining and am still having a similar issue on nearly every command:

channel 3: open failed: administratively prohibited: open failed
channel 5: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed
channel 3: open failed: administratively prohibited: open failed

[alokedesai]

@orionarcher Is this a recent regression? Are you able to reproduce with this build?

[orionarcher]

I was able to reproduce with that build. I didn't see any option for syntax highlighting or error underlining present though.

[alokedesai]

@orionarcher Ok good to know, how long has this issue existed in Warp for you?

[orionarcher]

I only started using Warp a couple weeks ago and it happened the first time I sshed into a cluster.

[taman0753]

@taman0753 Are you able to reproduce this if you turn off syntax highlighting and error underlining?

Yes @alokedesai Issue is still there

[taman0753]

@alokedesai Can we get the update to correct this issue.

[horacimacias]

I'm having the same issue as well. In case this helps, in my case I'm using ssh controlmaster and I've seen this problem in a "child" ssh connection. I checked MaxSessions as suggested but I have that commented out. If there is any troubleshooting/logs that can help here just let me know; I'm eager to using warp more. keep up the good work!

[rohitrp]

Facing same issue. This started happening after I had updated Warp 2-3 weeks ago. Any update on this?

[alokedesai]

@taman0753 @horacimacias @rohitrp We're still trying to better understand why this is happening as we've been unable to reproduce on our end, thank you for your patience here.

In the meantime, it would be very useful for us if you could help us debug when this issue went wrong by testing on prior builds of Warp. Does this issue repro on this version of Warp?

[rohitrp]

@alokedesai I have been using this build (v0.2022.09.07.14.56.stable_00) for the last 1 hour and didn't face the SSH issue. With the latest build, I start getting the SSH errors (channel open failed) within mins. So, it looks like this build doesn't have this SSH related bug.

[dannyneira]

Hi @taman0753 @orionarcher @horacimacias, I'm working to reproduce this issue, and I just wanted to rule out if there are some unsupported plugins on the remote machines. Please try the following on the remote machines and see if the issues persist.

1. Re-enable the ssh wrapper in Warp.
2. Disable your dotfiles just for Warp on the remote machine by using this conditional statement:

if [[ $TERM_PROGRAM != "WarpTerminal" ]]; then
##### WHAT YOU WANT TO DISABLE FOR WARP - BELOW

# The entire contents of your bashrc/zshrc on the remote machine.

##### WHAT YOU WANT TO DISABLE FOR WARP - ABOVE
fi

This way we can make sure the plugins on the remote machine are disabled for Warp and not the cause of the issues.

[rohitrp]

@dannyneira I updated Wrap to the latest version and enabled SSH wrapper. Instead of having contents of .zshrc inside if statement, I just deleted (technically moved) my .zshrc file and still saw the following errors -

channel 19: open failed: administratively prohibited: open failed
channel 21: open failed: administratively prohibited: open failed
channel 13: open failed: administratively prohibited: open failed
channel 9: open failed: administratively prohibited: open failed
channel 21: open failed: administratively prohibited: open failed
channel 7: open failed: administratively prohibited: open failed
channel 13: open failed: administratively prohibited: open failed

So, hopefully this confirms that this is not due to any plugin.

[dannyneira]

Thanks for confirming @rohitrp! we may need to look into narrowing down the exact version which it started to fail.

[orionarcher]

I am also not using any external plugins, just a clean install of Warp. EDIT: (oop misunderstood)

[taman0753]

@taman0753 @horacimacias @rohitrp We're still trying to better understand why this is happening as we've been unable to reproduce on our end, thank you for your patience here.

In the meantime, it would be very useful for us if you could help us debug when this issue went wrong by testing on prior builds of Warp. Does this issue repro on this version of Warp?

@alokedesai @dannyneira This build is working fine for me.

[dannyneira]

@taman0753 @horacimacias @rohitrp We're still trying to better understand why this is happening as we've been unable to reproduce on our end, thank you for your patience here. In the meantime, it would be very useful for us if you could help us debug when this issue went wrong by testing on prior builds of Warp. Does this issue repro on this version of Warp?

@alokedesai @dannyneira This build is working fine for me.

Thanks for confirming, would you be available for a zoom session to go over this issue with our engineer to troubleshoot? Look out for an email from me.

[taman0753]

@alokedesai Any update here

[alokedesai]

@taman0753 We'd love to schedule a zoom session to help debug the issue further, see @dannyneira's comment above

[ephracis]

I can confirm that I get this on a newly installed RHEL 9.1, minimal installation. However, I just created a new tab and did a second SSH session to the same server and there I cannot seem to reproduce the issue.

I have a very vanilla Warp as well, just installed it a couple of days ago for the first time and haven't gotten around to do any customization.

My device is a Macbook Pro 13" with an M2 chip, if that makes any difference.

Again, not able to reproduce it anymore, though it was occurring all the time just an hour ago (why I bothered to find this thread).

[dilanx]

I'm having the same issue. Sometimes I get those messages for no channels, sometimes for one or two, and just now it was like this:

channel 21: open failed: connect failed: open failed
channel 23: open failed: connect failed: open failed
channel 25: open failed: connect failed: open failed
channel 27: open failed: connect failed: open failed
channel 29: open failed: connect failed: open failed
channel 31: open failed: connect failed: open failed
channel 33: open failed: connect failed: open failed
channel 35: open failed: connect failed: open failed
channel 37: open failed: connect failed: open failed
channel 39: open failed: connect failed: open failed
channel 21: open failed: connect failed: open failed
channel 23: open failed: connect failed: open failed
channel 21: open failed: connect failed: open failed
channel 23: open failed: connect failed: open failed
channel 25: open failed: connect failed: open failed
channel 27: open failed: connect failed: open failed
channel 29: open failed: connect failed: open failed
channel 31: open failed: connect failed: open failed
channel 33: open failed: connect failed: open failed
channel 35: open failed: connect failed: open failed
channel 21: open failed: connect failed: open failed
channel 23: open failed: connect failed: open failed
channel 25: open failed: connect failed: open failed
channel 27: open failed: connect failed: open failed
channel 29: open failed: connect failed: open failed
channel 31: open failed: connect failed: open failed
channel 33: open failed: connect failed: open failed
channel 35: open failed: connect failed: open failed
channel 37: open failed: connect failed: open failed
channel 39: open failed: connect failed: open failed
channel 41: open failed: connect failed: open failed
channel 43: open failed: connect failed: open failed
channel 45: open failed: connect failed: open failed
channel 47: open failed: connect failed: open failed
channel 49: open failed: connect failed: open failed
channel 21: open failed: connect failed: open failed
channel 21: open failed: connect failed: open failed
channel 23: open failed: connect failed: open failed
channel 25: open failed: connect failed: open failed
channel 27: open failed: connect failed: open failed
channel 29: open failed: connect failed: open failed
channel 31: open failed: connect failed: open failed

The spam glitched out the block leaving me unable to highlight things in the terminal, so I'll be disabling the wrapper until this is fixed.

[alokedesai]

We have a repro for this and are actively working on a fix. Expecting this to be out next week. Thank you all for your patience here!

[alokedesai]

In the meantime--another short-term fix for this would be to disable "completions as you type" and "syntax highlighting for commands". This should prevent the underlying issue from occurring when you SSH as we work on the real fix

[orionarcher]

Oddly, I downloaded the latest version of Warp and am no longer having problems. Just wanted to report.

[taman0753]

@alokedesai Is the fix released , can you share the build link here. TIA

[alokedesai]

Not yet, we are aiming to have this released this Friday (12/9). In the meantime if you turn off "completions as you type", and "syntax highlighting for commands", you will likely no longer see these issues occurring

[vogelbot]

Just chiming in... same issue for me. Bit of a dealbreaker unfortunately. Back to iTerm until Friday. 🙏🏼

[szgupta]

Hi folks, we believe we have a fix for this that we want to get out today. For anyone experiencing this issue on the latest release, we'd love for you to test out a new build with the fix! Would anyone like to volunteer? If so, please email me @ suraj@warp.dev (I'll reach out to the first few folks)

Again, thanks you all for your help and patience with this ❤️

[szgupta]

@taman0753 @vogelbot @rohitrp @horacimacias sorry for the extra ping, but would any of you be available anytime soon to test this out with us?

[vogelbot]

@szgupta Yeah I'd be happy to. Emailing now...

[szgupta]

Hey folks, thanks to those who reached out to help test the fix!

We just released version v0.2022.11.29.08.03.stable_07 which should include a mitigation to the problem. Please let me know if you're still able to reproduce the problem.

[horacimacias]

thanks. I just downloaded it and when I ssh to a particular host I see "Seems like your completions are not working..." and when running something like ls -la I do see channel 6: open failed: connect failed: open failed:

I have the following on the server's sshd_config:

Protocol 2
PermitRootLogin no
ChallengeResponseAuthentication yes
ClientAliveInterval 60
ClientAliveCountMax 2
Compression no
LogLevel VERBOSE
MaxAuthTries 3
MaxSessions 2
TCPKeepAlive no
AllowAgentForwarding no
Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
HostKeyAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss
KexAlgorithms ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256
MACs hmac-sha2-256,hmac-sha2-512,hmac-sha1
UsePAM yes
UseDNS no
GSSAPIAuthentication no
X11Forwarding no
X11UseLocalhost no
AllowTCPForwarding no
PrintMotd no
AcceptEnv LANG LC_*
Subsystem       sftp    /usr/libexec/openssh/sftp-server

is there a particular setting there affecting the completions?