SSH connection no longer working with connection muxing if master is already up

[GoogleCodeExporter]

What steps will reproduce the problem?
1. configure an ssh connection as described below
2. attempt to use with an existing master connection already up (if there is 
not a connection already established, it will connect just fine.)

What is the expected output? What do you see instead?
It should just connect like it always has. :) Instead it fails with: "The SSH 
Tunnel has unexpectedly closed." the details reveal nothing much:

OpenSSH_5.2p1, OpenSSL 0.9.8l 5 Nov 2009
debug1: Reading configuration data /Users/cabbey/.ssh/config
debug1: Applying options for athena
debug1: Reading configuration data /etc/ssh_config
debug1: auto-mux: Trying existing master

What version of Sequel Pro are you using? What version of MySQL are you
connecting to on the server?
0.9.9 build 3348

Please provide any additional information below.
my ssh connection configuration details are as follows:
mysql host 127.0.0.1
username/password/database/port: set appropriately
ssh host: athena
ssh user: blank
ssh password: blank
ssh port: grey 'optional' i.e. not set

In my .ssh/config, the stanza for athena looks like this:

host athena
    IdentityFile ~/.ssh/cab.smug
    HostName name.my.vpn.dns.knows.here
    HostKeyAlias athena
    ControlPath /private/tmp/cabbey-athena.socket
    ControlMaster auto
    ForwardAgent no
    ForwardX11 no
    Compression yes
    Cipher Blowfish
    ServerAliveInterval 30
    Protocol 2

Original issue reported on code.google.com by the.orig...@gmail.com on 1 Jul 2011 at 10:35

[GoogleCodeExporter]

Hi there,

Thanks for reporting this issue.  I must admit I don't know a huge amount about 
the whole master connection setup, but AFAIK there are a few nightly testers 
using it with success - I'd like to be able to fix this though.

The change we made since 0.9.8.1 was that instead of specifying -M (to always 
trigger master mode, I believe) we now specify -o ControlMaster=auto.  It 
sounds like it's using this setting, and trying to use the existing master - 
but presumably failing because of the custom ControlPath.

As we're not specifying a ControlPath setting, I believe we should just pick up 
the standard details from your .ssh config.  Any ideas why this might not be 
working correctly?

Original comment by rowanb@gmail.com on 1 Jul 2011 at 10:45

[GoogleCodeExporter]

I'd be happy to try to help you debug this... is there a switch I can throw 
somewhere to get the full ssh verbose debug put into the details window? Or see 
the exact ssh command you're execing?

Original comment by the.orig...@gmail.com on 1 Jul 2011 at 10:50

[GoogleCodeExporter]

That is already the "verbose" output, but it can be cranked up to 3*verbose :)

I'd be happy to send you a build with that set, but that might have to wait 
until Sunday.  In the meantime, the arguments to the command can be seen at 
http://code.google.com/p/sequel-pro/source/browse/trunk/Source/SPSSHTunnel.m#291
 .

More on this later - thanks again :)r

Original comment by rowanb@gmail.com on 1 Jul 2011 at 10:56

[GoogleCodeExporter]

Original comment by stuart02 on 4 Jul 2011 at 4:53

• Changed title: SSH connection no longer working with connection muxing if master is already up

[GoogleCodeExporter]

I'm running into the same issue with a similar setup. I'd be happy to help 
debug as well.

Original comment by mich...@zimride.com on 9 Jul 2011 at 12:36

[GoogleCodeExporter]

seeing same issue, doesnt seem to matter if master is up or not or if a master 
connection has even been created

 - if master has been created and up or if master not up:
OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /Users/anthonysomerset/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Requesting local forward LOCALHOST:52237 -> 127.0.0.1:3306
debug1: mux_client_request_session: master session id: 14
stdin: is not a tty
debug1: mux_client_request_session: master session id: 14

if master hasnt been created yet - screenshotted

Original comment by anth...@somersettechsolutions.co.uk on 17 Aug 2011 at 9:12

Attachments:

• [Screen Shot 2011-08-17 at 10.12.20.jpg](https://storage.googleapis.com/google-code-attachments/sequel-pro/issue-1095/comment-6/Screen Shot 2011-08-17 at 10.12.20.jpg)

[GoogleCodeExporter]

upgraded to latest nightly build to test, it now will make an ssh connection 
but fail on mysql connection if master didnt get setup, but now wont connect if 
master already up or created but not up

Original comment by anth...@somersettechsolutions.co.uk on 19 Aug 2011 at 2:43

[GoogleCodeExporter]

Original comment by schlabbe...@gmail.com on 21 Aug 2011 at 6:59

• Changed state: Accepted
• Added labels: Component-Logic, Priority-Low, Type-Defect, Usability

[GoogleCodeExporter]

I have committed a small change in connection muxing compatibility in r3406; I 
would appreciate feedback.  We'll be including this change as part of 0.9.9.1 
to attempt to improve the issue, as it's solely a detection issue and so at 
least won't make the situation worse!

Original comment by rowanb@gmail.com on 1 Sep 2011 at 12:08

[GoogleCodeExporter]

Doesn't seem to have made it any better though.

OpenSSH_5.2p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /Users/cabbey/.ssh/config
debug1: Applying options for athena
debug1: Reading configuration data /etc/ssh_config
debug1: auto-mux: Trying existing master

Original comment by the.orig...@gmail.com on 22 Sep 2011 at 1:24

[GoogleCodeExporter]

I cranked up the debugging on the master session... here's what it's logging 
when we try to start a new slave session:

debug1: fd 9 clearing O_NONBLOCK
debug3: ssh_msg_recv entering
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug3: muxserver_accept_control: receiving 0 env vars
debug2: muxserver_accept_control: accepted tty 0, subsys 0, cmd 

One difference here... a normal shell, the accepted tty value is 1. I assume 
you're connecting with -T.

debug2: muxserver_accept_control: got fds stdin 10, stdout 11, stderr 12
debug3: ssh_msg_send: type 2
debug3: fd 10 is O_NONBLOCK
debug2: fd 11 setting O_NONBLOCK
debug2: fd 12 setting O_NONBLOCK
debug2: fd 9 setting O_NONBLOCK
debug1: channel 1: new [client-session]
debug3: muxserver_accept_control: channel_new: 1
debug2: channel 1: send open
debug2: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 1: request auth-agent-req@openssh.com confirm 0
debug2: client_session2_setup: id 1
debug2: channel 1: request shell confirm 1
debug2: callback done
debug2: channel 1: open confirm rwindow 0 rmax 32768
debug2: channel 1: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 1
debug2: shell request accepted on channel 1

This is where it stops being normal.... everything before now is the same as 
what is seen for a normal slave startup.

debug2: channel 1: read<=0 rfd 10 len 0
debug2: channel 1: read failed
debug2: channel 1: close_read
debug2: channel 1: input open -> drain
debug2: channel 1: ibuf empty
debug2: channel 1: send eof
debug2: channel 1: input drain -> closed
debug2: channel 1: rcvd eof
debug2: channel 1: output open -> drain
debug2: channel 1: obuf empty
debug2: channel 1: close_write
debug2: channel 1: output drain -> closed
debug1: client_input_channel_req: channel 1 rtype exit-status reply 0
debug2: channel 1: rcvd close
debug3: channel 1: will not send data after close
debug2: channel 1: send close
debug2: channel 1: is dead
debug2: channel 1: garbage collecting
debug1: channel 1: free: client-session, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r0 i0/0 o0/0 fd 5/6 cfd -1)
  #1 client-session (t4 r2 i3/0 o3/0 fd -1/-1 cfd 9)

debug3: channel 1: close_fds r -1 w -1 e 12 c 9

Is there anyway to see the command being sent? I'd like to simulate the same 
command with the command line ssh client.

Original comment by cab...@cabbey.net on 22 Sep 2011 at 4:02

[GoogleCodeExporter]

cabbey,

That's very helpful, especially as it's now working in my (rather limited) test 
setup!

The SSH command being sent can be approximated with:

/usr/bin/ssh -N -v -o ControlMaster=auto -o ExitOnForwardFailure=yes -o 
ConnectTimeout=10 -o NumberOfPasswordPrompts=3 -o TCPKeepAlive=no -o 
ServerAliveInterval=60 -o ServerAliveCountMax=1 -p SSHPORT SSHUSER@SSHHOST -L 
RANDOMLYGENERATEDFREEPORT/127.0.0.1/MYSQLPORT

There's some additional arguments added for keys; an environment is also set up 
with the following values:
SSH_ASKPASS - a password helper app for the connection
DISPLAY - :0

I think that's it...

Original comment by rowanb@gmail.com on 22 Sep 2011 at 4:12

[GoogleCodeExporter]

hmm... so I pulled the current code from svn (r3428) and put a break point at 
the point the task is launched in SPSSHTunnel::launchTask(). when it hit that 
break point I dumped the full argument list and came up with this command line:

-N -v -o ControlMaster=auto -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o 
NumberOfPasswordPrompts=3 -o TCPKeepAlive=no -o ServerAliveInterval=60 -o 
ServerAliveCountMax=1 athena -L 65157/127.0.0.1/3306

(athena is the name of the host stanza in my ssh_config)

When I run that ssh invocation from the command line alone it works fine. I get 
a shell on the remote server with a bash instance sitting there waiting for me 
to do something.

Next I dumped the environment it runs it (much longer than the one you noted, 
but not really interesting) and ran the ssh command under env to get just that 
env:

env -i "Apple_PubSub_Socket_Render"="/tmp/launch-eFf02H/Render" COLUMNS=80 
"COMMAND_MODE"=unix2003 "CVS_RSH"=ssh DISPLAY=":0" 
"DYLD_FRAMEWORK_PATH"="/Users/cabbey/src/sequel-pro-read-only/build/Debug" 
"DYLD_LIBRARY_PATH"="/Users/cabbey/src/sequel-pro-read-only/build/Debug" 
"DYLD_NEW_LOCAL_SHARED_REGIONS"=YES "DYLD_NO_FIX_PREBINDING"=YES 
HOME="/Users/cabbey" LINES=24 LOGNAME=cabbey NSUnbufferedIO=YES 
PATH="/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin" 
PWD="/Users/cabbey/src/sequel-pro-read-only/build/Debug" SHELL="/bin/bash" 
SHLVL=0 "SP_CONNECTION_NAME"="SequelPro-2231988677647579780" 
"SP_CONNECTION_VERIFY_HASH"=9627605507218486304 "SP_PASSWORD_METHOD"=2 
"SSH_ASKPASS"="/Users/cabbey/src/sequel-pro-read-only/build/Debug/Sequel 
Pro.app/Contents/Resources/SequelProTunnelAssistant" 
"SSH_AUTH_SOCK"="/tmp/launch-GPAOAJ/Listeners" 
TMPDIR="/var/folders/7j/7jzmkqAGHmK1FwMIY08wy++++TI/-Tmp-/" USER=cabbey 
USERBREAK=1 "__CF_USER_TEXT_ENCODING"="0x1F5:0:0" /usr/bin/ssh -N -v -v -v -o 
ControlMaster=auto -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o 
NumberOfPasswordPrompts=3 -o TCPKeepAlive=no -o ServerAliveInterval=60 -o 
ServerAliveCountMax=1 athena -L 65157/127.0.0.1/3306

Same deal.... it works just fine, I end up with a bash session on my db host 
waiting for input.

that said, running under xcode, I can't make sequelpro connect to anything over 
ssh... it just spins and times out, I assume there's a trick to debugging it, 
since I see some comments in the source about various things not working 
correctly due to the parent of the task being xcode instead of sequelpro.

Original comment by cab...@cabbey.net on 23 Sep 2011 at 4:29

[GoogleCodeExporter]

(I should point out, to avoid confusion, that cab...@cabbey.net and 
the.orig...@gmail.com are both me... just google failing at single signon.)

Original comment by the.orig...@gmail.com on 23 Sep 2011 at 4:32

[GoogleCodeExporter]

This issue can be resolved by adding:
-o ControlPath=none

This'll tell ssh not to use the existing ControlMaster, as I don't think 
openssh will let multiplexed session add port forwards.  This is a bit of an 
assumption on my part, based on the following:
Multiplexed connection:
[jspelman@SERVER ~]$ ~?
Supported escape sequences:
  ~.  - terminate session
  ~B  - send a BREAK to the remote system
  ~R  - Request rekey (SSH protocol 2 only)
  ~#  - list forwarded connections
  ~?  - this message
  ~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

From the session running the ControlMaster:
[jspelman@SERVER ~]$ ~?
Supported escape sequences:
  ~.  - terminate connection (and any multiplexed sessions)
  ~B  - send a BREAK to the remote system
  ~C  - open a command line
  ~R  - Request rekey (SSH protocol 2 only)
  ~^Z - suspend ssh
  ~#  - list forwarded connections
  ~&  - background ssh (when waiting for connections to terminate)
  ~?  - this message
  ~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

~C is how you control port forwards for a live session, and since I can't find 
details in the docs, I'm basing my assumption on the above

As a workaround, I have entries in my .ssh/config file that specify 
ControlPath=none and have SequelPro using those entries

Original comment by jasiel.s...@gmail.com on 19 Dec 2011 at 10:35

[GoogleCodeExporter]

Jasiel, thank you for providing the key to this - I'm sorry I missed it, my 
only excuse is that I was on honeymoon at the time :)

It does appear that the issue is that certain combinations of SSH servers and 
clients don't allow a port forward to be added to a live session.  Oddly some 
definitely do, which seems to go against what you'd expect... in additional, if 
Sequel Pro creates a connection mux, this could cause problem for interactive 
slaves using that same connection thereafter.

One fix for this is definitely to disable connection muxing, as Jasiel 
suggests.  What I've done instead in r3706 and r3707 is to use a custom path 
within Sequel Pro, ensuring it's not shared with normal interactive sessions.  
This way Sequel Pro still gets the benefit of connection sharing within the 
app, but doesn't break - and isn't broken by - interactive sessions.

Sorry that this took so long to sort out :)  Thanks all for your input!

Original comment by rowanb@gmail.com on 28 Jun 2012 at 11:41

• Changed state: Fixed

[GoogleCodeExporter]

Been testing this with a nightly build,  great solution Rowan... can't wait to 
see this land in a new official build. (not to mention the slick new icon.)

Oh, and congrats on the wedding. :)

Original comment by the.orig...@gmail.com on 2 Aug 2012 at 9:59

[GoogleCodeExporter]

For a quick work-around just edit (or create) your SSH config file:

~/.ssh/config

Add the following directives:

ControlMaster auto
ControlPath /tmp/ssh_mux_%h_%p_%r

Then restart SequelPro. After connecting a few times you may start seeing the 
error mentioned above. If that is the case then all you need to do is delete 
any temporary SSH session files in /tmp:

rm /tmp/ssh_mux*

Of course, this will also destroy sessions created by other SSH connections. 
But at least it seems to work.

Original comment by j...@nolaninteractive.com on 13 Nov 2013 at 5:10