SSH Agent Forwarding

[dt]

Would be great to be able to forward local SSH agents. Seems to ignore ForwardAgent yes in ~/.ssh/config

[jlynn]

:+1: I can't use mosh for this very reason. I need to be able to connect to other machines in the environment I'm accessing plus use my ssh key to push/pull code from github.

[peterjeremy]

This requires tunnelling the agent request from the server back to the client.

[phmarek]

I'd be interested in this feature, too.

[dt]

If #41 ends up being resolved be adding tunnels for arbitrary octet streams, agent forwarding should be trivial to add on top of that.

[thrillcall]

+1 for being able to ForwardAgent. In our case also critical for getting code to/from GitHub on remote servers.

[dbanck]

+1 I need the agent for sudo authentication.

[clneagu]

+1 mostly for github

[avleen]

+1 also!

[stianeikeland]

:+1: .. this would be very useful

[nostra]

+1

[plathrop]

+1

[plathrop]

Hard to resist the urge to say nothing but '+1' to @Mkaysi's comment...

[keithw]

All right, folks, the message has been heard loud and clear! We will put this on the roadmap for a release, probably the one immediately after 1.2. No need to keep +1ing - we will do it.

[Hidendra]

Any word on if this is still planned for 1.2 or near/after 1.3?

[chekhovs1]

+1 from me. I naively thought that this was implemented and after building mosh at the server systems and running it, realized that this is not quite ready yet. Any updates on the timeline?

[kergoth]

+1 for me as well, this is a blocking issue for me, as I do remote development, using local keys to communicate with upstream SCM repositories.

[ogrisel]

+1 mostly for being able to work with github when doing remote development from EC2 machines.

[tsuna]

Sorry for spamming everyone, GitHub doesn't appear to have a feature to get notifications of changes on an issue without +1'ing it. So +1 for me too.

[andreypopp]

@tsuna see at the bottom of any issue page

[Gonzih]

+1 for agent forwarding

[ghost]

Came to report another issue, saw this. Key forwarding is absolutely required to my workflow and I am seriously concerned about the way the mosh devs handle ssh keys if it wasn't baked in from the beginning.

[keithw]

Mosh works fine with SSH keys (or any SSH login method), since we just use SSH itself to handle authentication.

We don't allow the user to forward ssh-agent requests because, frankly, nobody has contributed the code and we haven't gotten around to it.

But in general, ssh-agent forwarding is inappropriate for high-security environments. There is little to stop a compromised remote host from requesting your credentials to do 'rm -rf /' (or modify authorized_keys) on a bunch of remote hosts. The user doesn't find out what the remote host wants to do with your credentials; the ssh-agent protocol just lets a process use your key to sign any request. If you forward ssh-agent, then it lets a remote host use your key to sign any request. It's almost the same security characteristics as just copying your credentials to the remote host in the first place -- but not quite, since the remote host loses access to your credentials after you log out (unless it uses them in the meantime to insert its evil key in your authorized_hosts file elsewhere).

I am surprised by all the people saying we need this because they are security freaks. You don't forward ssh-agent if you are a security freak. ssh-agent forwarding is a convenience feature that should not be on by default and should only be used when connecting to semi-trusted hosts.

@apeiron, no need for concern. A process that does ssh-agent forwarding just takes bytes from the remote host and ferries them to the local host's ssh-agent and similarly for the reply. It doesn't touch secret data. All it does is make remote requests to the ssh-agent look like they are coming from the local host. For the reasons above, that's a risky thing to allow, but actually making it happen is easy.

[maxcom]

@keithw ssh-agent has confirmation feature that can protect you from that scenario when remote host does 'rm -rf /' on other remotes.

ssh agent forwarding is really good for accessing git repository from development and testing servers. Also it is good for proxying ssh connections through other hosts when there is no direct connection.

[keithw]

Tell me more about this confirmation feature. My understanding is that there is no way for the agent (and therefore the local user) to learn what command the remote host intends on executing once it is able to authenticate. All the protocol allows is for ferrying public-key challenges and the associated responses.

[maxcom]

Yes, you can not controll what command is executed. But you can control what hosts are involved. 31.10.2012 9:54 пользователь "Keith Winstein" notifications@github.com написал:

Tell me more about this confirmation feature. My understanding is that there is no way for the agent (and therefore the local user) to learn what command the remote host intends on executing once it is able to authenticate. All the protocol allows is for ferrying public-key challenges and the associated responses.

— Reply to this email directly or view it on GitHubhttps://github.com/keithw/mosh/issues/120#issuecomment-9933956.

[keithw]

I don't think that's right. The ssh-agent only learns which identity (key pair) the requesting process is trying to use. It doesn't learn which host the request came from (remember, forwarding makes the request appear to be coming from the local machine) and it doesn't learn which remote server that process is trying to connect to, or what command the process is trying to execute on the remote server. Happy to be corrected though if my understanding is off.

[maxcom]

Yes, it seems that you are right. So the better way to improve ssh-agent security is using different keys for different groups of servers. Anyway, that is generic weakness of single sing on systems. SSH has agent forwarding feature and it will be nice to have it in mosh.

[tsuna]

I think for most people waiting on this issue, this isn't as much about conveniency as it is about security.

[rinne]

But with ssh you can select which connections have agent forwarding and which not. So if you log into the system you don't trust, you shouldn't forward your agent to this. I don't know if this is the case with OpenSSH, but at least the commercial one works so that for agent forward connections, ssh server adds a "forwarding identifier" so that the agent itself is able to see over what kind of path the forward is opened. Even if the system at the end of the path is compromised, it can also forge the end of the path, not the path that leads to it from the client.

[rinne]

And yes, agent forwarding for mosh is a must! Much more important than X11 or generic tcp stream forwarding. It's also something that is not particularly performance critical, since agent connections don't happen all the time and individual sessions are typically in the magnitude of kilobyte or less.

[blueben]

Indeed. While I appreciate the hard work that goes in to the project, I don't think it is appropriate for the mosh project to determine which risks its users should find acceptable. If SSH supports it, mosh should support it. In this case, the scenario you are concerned about is most often found on bastion hosts to access machines which are otherwise inaccessible. If the user doesn't use ssh-agent forwarding, then they need to keep keys on the bastion host, which results in the exact same vulnerability should that host be compromised. There is no improvement in security simply because mosh doesn't support the feature. If I need additional security measures for my keys, I'll make sure they have passphrases or I'll use multi-factor auth with a FOB.

Put bluntly, mosh loses users because this feature doesn't exist. I can only use it in a few of my active terminals, holding out hope that I will eventually be able to use it for all my workflows when this feature is added. I evangelize mosh quite often and loudly, but the lack of agent forwarding is a common complaint. I evangelized to apeiron and he wants the feature badly enough that he came here to complain, so QED I suppose.

And yes, this feature is far more important than X11 or stream forwarding. I'm willing to bet your best and most common users are IT pros. This feature is far more desirable to them than these other features.

[keithw]

Again, the primary reason we don't have ssh-agent forwarding is that nobody has contributed the code.

But I don't agree that Mosh should necessarily support insecure protocols just because SSH does. The idea of Mosh is to improve on SSH, and one way is by having a simpler and more robust security story. SSH's many security holes have largely been because of its complexity and the large number of options (including letting you negotiate weak ciphers and weak protocols, leading to downgrade attacks). I don't think we want to follow SSH down every dark path.

For bastion hosts where the problem is connectivity, I (1) agree that copying and forwarding credentials have similar security risks (this is why I think it's silly all the people who think ssh-agent forwarding is so much more secure) but (2) disagree that either of these behaviors is justified.

The right approach is for the bastion host simply to proxy the (encrypted, authenticated) connection to the other machines, and to make the authenticated connection end-to-end. There is no need to trust the bastion host at all or give it credentials -- we are just using it as a network proxy. We can make Mosh (or socat) do this and it will be more secure than ssh-agent forwarding.

[treed]

For the bastion host problem in particular, this is one option (with ssh) on how to achieve that without agent forwarding:

host hidden ProxyCommand ssh bastionhost nc -q 5 %h 22

OTOH, I agree that agent forwarding is pretty convenient for other cases. For me personally, what really keeps me from being able to use mosh at work is the fact that I need to open up new firewall holes. But I wouldn't say that agent forwarding is good for security.

[blueben]

And again, it has not been demonstrated that agent forwarding is inherently insecure to such a point that it should not be implemented in mosh. Further, as I pointed out above, any problems are easily mitigated with key passphrases and multi-factor auth. In addition, bastion hosts are well understood and oft used, usually as more than just a network proxy. In many cases, it's used as an intentional and auditable choke-point with tools which folks do not want installed on other servers. In other words, direct end-to-end connections are exactly what those environments are trying to avoid.

Let's not try and throw away everyone's workflows because we can't imagine the needs of other environments or they don't fit into one specific mindset. If we begin our argument with "the only right way is", we either know everything about every possible environment or we're fooling ourselves. If mosh can implement another, better way to directly reach hosts behind network choke points, that's great. But that does not make it unnecessary to implement this feature.

[martinrehfeld]

IMHO, it would make sense focusing on finding someone that actually can implement agent forwarding (maybe as a sponsored feature?). Arguing over it does not make the need go away. It's a bit of a shame that mosh cannot be enjoyed in a lot (dare I say: most) use-cases.

[romaninsh]

Please provide instructions how to support second host authentication without storing the key on intermediate server. If forwarding is undesired, what's the proper alternative?

[blueben]

@romaninsh It has been proposed that nobody use jump servers, but instead directly connect to everything everywhere. I suspect that anyone who proposes this may not realize that this is not always possible, and in fact a violation of a number of security protocols in many environments.

[rinne]

@romanish I don't think there is such alternative. If you don't count using password in second authentication :).

I'd say that need to use git, rsync, scp, et cetera is full justification to have agent forwarding. Those are often needed even if intermediate hosts are not used in logging to intranet hosts.

[keithw]

Using a jump server is fine. Trusting a jump server with the plaintext of your communications, and the ability to execute arbitrary commands, is unnecessary if it's really just a jump server.

[blueben]

I'll make my own trust decisions, thanks.

[keithw]

@romanish: The "proper" alternative would probably be a protocol that let you authorize a host to perform some operations (e.g. to do a git push to github for a particular repository) using your credentials, in revocable and tracable fashion, without giving away the store.

The problem is that ssh-agent was not intended to be forwarded. The ssh-agent can't tell the difference between a local request and a remote one, and it can't tell who is making the request or what they want to do with your credentials. So there's not really enough information in the protocol to make sane security decisions. In most cases it's basically better to actually store a private key on the remote server itself (and give that identity limited authorizations), rather than blindly forward requests from the remote server back to your ssh-agent.

[blueben]

A private key store on the remote server. Did you just try to solve a perceived security issue by adding more complexity? LOL

Come on, as security risks go, this one is far to the bottom of an extremely long list. While it is theoretically possible, the fact remains that if your jump/gateway server is compromised such that an attacker can request your keys then you've already lost the game and not-forwarding does absolutely nothing to improve your posture at that point.

[blueben]

Addendum after some more contemplation.

The issue here is that you're trying to solve the wrong problem. The keys are not what is important to defend against an attacker. What we actually want to defend is access to other machines. In that context SSH-forwarding is not the actual problem, but rather the use of keys without passphrases and hosts which allow access without multi-factor challenges. You can't solve the problem of poor key hygiene by making keys even more difficult and complex to manage.

[tsuna]

This issue is about having some support for SSH agent forwarding with Mosh. Could we please not turn this issue into a debate on whether this is more or less secure than another alternative, or which way is the best?

[romaninsh]

@keithw I understand your reasoning. Any rogue app on any host you log into can potentially request as many authentications as it wants without me noticing. (http://serverfault.com/questions/112920/how-can-i-activate-ssh-agent-confirmation-in-macosx-leopard)

It's also clear that forwarding octets alongside your connection is a no brainer for many and is essential feature. Clearly we are forwarding keystrokes all the way to the server, why only keystrokes.

I appreciate your effort to solve the problem with agent but it has nothing to do with mosh and by avoiding this feature you will alienate your users until eventually someone would fork your code and add the feature.

My suggestion would be to add this feature but put a big warning when it's used prompting user to secure their ssh-agent. As the popularity of "mosh" grows your message will make more people aware of the problem and would make it easier to come up with a solution.

Unfortunately I am in a whole different development universe, otherwise I would have helped you with the patch.

[keithw]

Thanks romaninsh. Yes, I think we will support ssh-agent forwarding with a flag like --unsafe-agent-forward, and we will also notify the user (with a popup message in the blue bar) when an agent request gets made. I suppose we could also just let the user turn forwarding on and off at runtime with a key combination.

I also would like to support gateway connections, with a syntax like mosh --via jumpserver remotehost, that don't trust the gateway host (any more than they trust any random gateway along the network path).

That's the plan, at least. Anybody who wants to get involved is welcome to contribute and join in on the mailing list or IRC channel.

[romaninsh]

@keithw Great decision! Me and everyone else really appreciate what you are doing, thanks for your patience.

[mbelgin]

The lack of ssh-agent is a show stopper for Mosh... Thanks for your considering to support it.

[rinne]

I must say that using language like unsafe-agent-forward in command line flags is quite misleading. I accept that agent forwarding should be something that needs to be explicitly enabled either by command line option or an item in a configuration file, because it enables additional way to authenticate and it may not be something that all users want and sometimes the user may want to log into a system that he doesn't trust, in which case forwarding agent connections from these systems is unwise.

A few points though:

1) If the user is running an authentication agent in his desktop machine, he must make two more or less conscious decisions in order to have his key to be visible in the remote host through agent connection. He must add the key to the agent and allow agent forwarding for ssh (or mosh once implemented) for the connection to that particular remote host.

2) The private authentication key can not be stolen via agent connection, but instead cryptographic operations (sign, decrypt) can be performed instead and they can be performed only while the connection with agent forwarding is open. In order for this being a real problem, it requires the system that is connected with agent forwarding enabled to have either compromised security or crooked administrator. While neither is unheard of, it would be much more damaging in a system like that to authenticate to another remote system using a password, which can be easily stored and used whenever the attacker wants until the password is changed or indefinitely, if the attacker installs some kind of backdoor using the password.

3) Same problem is in cases where the user is forced to store private authentication keys to remote hosts. In case these hosts are compromised, the key files can be stolen and passphrase to these keys can be snooped later or exposed to brute force password attacks or something more clever. The stealing of the key files and somehow acquiring the passphrase doesn't have to happen at the same time, so the attacker has time.

4) While forwarded agent can naturally also be used in installing backdoors just like the password, the time window for that is shorter and as said earlier, the user has control over which credentials are stored to the agent.

5) While standard agent in OpenSSH distribution doesn't have any method to require user confirmation for individual agent operations (i.e. requests to perform cryptographic operations on the private key stored to the agent) there is no reason why it must be so. Actually at least some agent implementations do just that and in addition to that also allow configuration per key to enable interactive verification for just some keys. Also other security features could be added to the agent. A couple ones intuitively coming to mind is to (add configuration to) reject agent operations if the desktop is idle or locked and allowing adding of keys without passphrase so that passphrase is asked only when the key is used for a first time and forgetting the passphrase if the key is not used within some configured period. I think that there are also some agent implementations that are doing just that (and honestly I've personally implemented one in past).

What I'm trying to say, is that mosh should have agent forwarding in order to be a viable solution for infinite number of administration and software development scenarions, and adding agent forwarding to mosh so that it's enabled by command line parameter or configuration file option is not a step backwards in security but clearly forwards, because it in most cases eliminates the need to store ssh private keys to remote hosts as well as use of passwords when logging to another remote host from these ones. "Logging to another remote host" in most cases mean using git or scp rather than actually opening an interactive shell connection to anywhere while ssh being just a vehicle for data transport and authentication.

I find it a bit peculiar that some mosh people have quite prejudiced attitude against the authentication methods in ssh, which is quite time proven, and at the same time mosh is using ssh for authentication and connection setup.

As said earlier, I volunteer to help in specifying the feature and provide with an analysis of security and listing the caveats, but I don't have time to familiarize myself with mosh implementation enough to be able to write the channel code required in the implementation.

However, OpenSSH is distributed with BSD-like license so at least the code to create agent listeners in a safe way (for proxy listeners opened by mosh server) and also connecting to ones (for mosh client to connect the local agent listener when receiving a forwarded agent connection via mosh connection) can be either grabbed from there or alternatively reimplemented using that as a security guideline.

[martinrehfeld]

+1 for cutting the --unsafe-agent-forward crap and using familiar ssh CLI switches, i.e. -A in this case. Either way, getting Agent Forwarding into mosh is the important part here.