ssh-agent: permanent loaded ssh-key storage

[mluypaert]

"OpenSSH for Windows" version 7.7.2.2

Server OperatingSystem Irrelevant

Client OperatingSystem Windows 10 Home (18362.418)

The problem I am using private SSH-keys encrypted with a passphrase. When using any SSH-agent, that means I have to provide my passphrase to be able to load the key and use it. I've used pageant before and there you load the key and provide the passphrase. Whenever pageant gets terminated (by shutdown, rebooting, ...) the key is removed from memory, and I have to provide my passphrase again the next time I want to use it. Every time, hence the security advantage of having an encrypted SSH key. With the windows openssh agent however, that's a different story. I have my agent set to automatically start with windows (I believe by running Set-Service ssh-agent -StartupType Automatic) just like I did with pageant. However, the windows openssh agent seems to permanently store my SSH key somewhere after loading it to the agent once (ssh-add), because when I run ssh-add l after a restart of my computer, they encrypted key which I loaded before the restart is automatically loaded again, without requiring my passphrase again. This completely voids the advantage of a passphrase-protected SSH key and to my opinion is a big security issue.

Proposed solution Do not store my SSH key permanently when I load it into the agent, but only temporarily store it in the agent's memory. Or at least store it in it's passphrase-encrypted form and ask for my passphrase every time the key gets loaded.

[alexandre-abrioux]

While waiting for this update, I created a scheduled task that clears the keys on boot. I'm sharing here the parameters I've used:

1. Open the Task Scheduler (Start Menu > Search "Task Scheduler")
2. Create a new task
3. For the trigger, use "At log on" and select your user account
4. For the action, use the following:
   • program: cmd.exe
   • arguments: /C "ssh-add -D"

I hope this can help others 🙂

[vixie]

On Tue Jan 17, 2023 at 11:07 AM UTC, Alexandre ABRIOUX wrote:

While waiting for this update, I managed to create a scheduled task. I'm sharing here the parameters I've used: ...

3. For the trigger, use "At log on" with your user account ...
   • arguments: /C "ssh-add -D"

I hope this helps 🙂

thanks. if i'm reading this right, it will clear the auth cache at login time. this leaves privileged secret information in the auth cache from the time of the last ssh-add to the next login, which could be overnight or as long as a vacation. if there's a way to trigger this at logout time rather than login time, and if logout can be triggered on a timeout if no user input occurs for some idle threshold period, it would be a good workaround.

for now i have deinstalled microsoft ssh, and am using msys2 tools. this had the side effect of getting a terminal application ("mintty") with working IRM. my UNIX side editors appreciate this.

[HiFiPhile]

For anyone interested, I've created a fork which the ssh-agent storage credentials in the traditional non-persist way and has telemetry removed: https://github.com/HiFiPhile/openssh-portable

PKCS11 is untested

[ioMatrix]

The latest version pageant is able to start with a --openssh-config option which will generate a named pipe usable by OpenSSH.

Details on how to enable are available here:

https://the.earth.li/%7Esgtatham/putty/0.78/htmldoc/Chapter9.html#pageant-cmdline-openssh

[J-Sorenson]

I discovered this issue within my company when we started moving from PuTTY to Microsoft's built-in OpenSSH implementation for our secure connections. We require that the SSH connections to have a separate passphrase so that if someone were to illicitly gain admin access to the PC, they still won't have access to the engineering network. I was hoping to work around this with the usual "-t" option, but apparently that is still not supported per issue #1056 from Feb 2018.

Putty Pageant drops the keys on logout. Linux ssh-agent drops the keys on logout. I don't think those design decisions were random, so permanently unlocking the keys is a perplexing "by design" decision that is counter to other implementations. We've implemented the "ssh-add -D" logout script in the group policy as a work-around, but that doesn't resolve a hard power-off situation.

We look forward to seeing ssh-agent using a more common and secure implementation in the near future.

[HiFiPhile]

@J-Sorenson

You can give a try of my fork if you want.

[J-Sorenson]

Thanks for the offer, but that won't be permitted on the corporate network. It needs to come from Microsoft or an approved 3rd-party software solution.

[vixie]

i'm using MSYS2 now which has a normal openssh and ssh-agent, plus a terminal application called UCRT64 that's very complete. added tmux and i'm in happy land. note that i'd prefer to use microsoft's ssh but with their insistence on keeping my unlocked key in persistent (across logout or reboot) storage is so completely wrong that i can't imagine letting it possess my keying material in any form.

[aservedio]

Personally since then I fully switched to working from within WSL and using ssh-agent within it. Event went as far as building a packaging system to build WSL images pre-setup with ssh-agent and gpg-agent.

Very Linux focused, so no integration with Windows at all. But it does work nicely with VSCode and devcontainers! :)

References: info on wsl.geekstuff.dev, example for ubuntu 22.04 with systemd

[tlartaud]

This issue is opened for years now. Is there an official fix now?

Also, I just wanted to share my opinion, from a newbie perspective. I am absolutely not expericenced as you all and cannot judge the technical aspects of the decisions made in Windows' OpenSSH. However, I think that the way the process is designed actually breaks the User Experience.

Like many users there, I was, at first usintg Putty, or even OpenSSH on Linux (or more formerly, Git for WIndows' agent), and I was expecting the same behaviors when switching to the Windows version. This means that, since more than a year, I am unnecessarily re-loading my keys through ssh-add before running a VM, or before attempting to connect to a remote server, and I am unnecessarily re-typing the passphrase again, almost every day, while my keys are already stored somewhere without my knowledge: ssh-add doesn't notice you that the key is already loaded. There are probably many users that reload their keys after a reboot, and they probably don't even know they do not need to do that. What's the point of permanently storing a key if the end-user doesn't know about it?

One day I tried to SSH to my remote, and I instantly realized I forgot to first ssh-add... But it worked... I was just shocked. At a first glance, I thought that my OpenSSH was buggy or corrupted (reboot scripts not being executed?), so I reinstalled it, and rebooted the computer. Once reinstalled, i did an ssh-add -L, and all the keys were still there, still loaded, even after a reinstallation of OpenSSH.

Well, this was really confusing.

[vixie]

This issue is opened for years now. Is there an official fix now? ...

so, i can't speak for microsoft, but i think ssh has been embraced and extended, in order to differentiate windows from other ssh-capable platforms. my prediction is they'll document what the behaviour is and that it is intentional, will not change their ssh client to make this optional or add a knob to turn it off, and that this ticket will remain open until long after we are all drinking mead together in valhalla. i hope i'm wrong about any part of that.

[J-Sorenson]

Well, I was rather hoping Microsoft had abandoned the old technique of "embrace, then conflict" like they did with IE6 web standards, or the XML document standard, or... Anyhow, our solution was to apply a "ssh-add -D" to the logout process. It's not perfect, but it gets us close enough for now.

[ahicks92]

it seems that this is now quite off topic, but MS has only slight control over the SSH protocol if any at all. Just because a specific implementation is different doesn't mean they're trying to take it over, and indeed they probably can't even if they wanted to. Not having this issue resolved is annoying, but I don't think it's fair to attack people over what is almost certainly just a priority decision.

[vixie]

Austin Hicks wrote on 2023-08-18 08:45:

it seems that this is now quite off topic, but ... but I don't think it's fair to attack people over what is almost certainly just a priority decision.

since you're objecting to things i did not say, and ignoring things i did say, i think we can let this drop now.

-- P Vixie

[metzz1]

While waiting for this update, I created a scheduled task that clears the keys on boot. I'm sharing here the parameters I've used:

1. Open the Task Scheduler (Start Menu > Search "Task Scheduler")
2. Create a new task
3. For the trigger, use "At log on" and select your user account

4. For the action, use the following:

   • program: cmd.exe
   • arguments: /C "ssh-add -D"

I hope this can help others 🙂

this solution works well, thanks for sharing :)

[mokeyish]

While waiting for this update, I created a scheduled task that clears the keys on boot. I'm sharing here the parameters I've used:

1. Open the Task Scheduler (Start Menu > Search "Task Scheduler")
2. Create a new task
3. For the trigger, use "At log on" and select your user account

4. For the action, use the following:

   • program: cmd.exe
   • arguments: /C "ssh-add -D"

I hope this can help others 🙂

this solution works well, thanks for sharing :)

It would be better if you could write a wrapper for ssh-add. When using ssh-add, a task is automatically created and delayed for 30 minutes to execute ssh-add -D.

[ynuwenhof]

The workaround for this issue that I used to remove all identities on shutdown:

1. Create a very simple PowerShell script with echo "ssh-add -D" > ssh-flush.ps1
2. Move the PowerShell script into the C:\Windows\System32\GroupPolicy\Machine\Scripts\Shutdown folder
3. Press WIN + R and enter gpedit.msc
4. Navigate to Local Computer Policy > Computer Configuration > Windows Settings > Scripts (Startup/Shutdown) > Shutdown
5. Click on the PowerShell Scripts tab and click add
6. Click browse next to the Script Name: text box and select the previously created and moved ssh-flush.ps1 script
7. Click OK and Apply

Now your SSH identities should be removed automatically every time you shut down Windows.

Optionally, the PowerShell script can also be executed on logoff instead of shutdown by navigating to User Configuration instead of Computer Configuration.

[vixie]

This is cool thanks. It still leaves a hole on power loss and panic and other unclean shutdowns but it is better than every account having to do this separately.

p vixie

[snowzach]

I am not sure if anyone else is in the same boat as me but the only reason I used SSH from Windows-Land in the first place is to use the Remote SSH plugin in VS Code. Otherwise all SSH'ing I do is from WSL where the agent works as expected. I just discovered if you want to force VSCode to use SSH from WSL instead of Windows and stop using Windows SSH and agent altogether you can:

Create a bat file somewhere like your home directory called C:\Users\YourUser\ssh.bat with the contents:

C:\Windows\system32\wsl.exe bash -ic 'ssh %*'

This forces it to run the login scripts which was important for me because that sets up the SSH agent inside of the WSL environment. I happen to use this in a bashrc to start the ssh-agent:

if [ -z "$SSH_AUTH_SOCK" ]; then
   # Check for a currently running instance of the agent
   RUNNING_AGENT="`ps -ax | grep 'ssh-agent -s' | grep -v grep | wc -l | tr -d '[:space:]'`"
   if [ "$RUNNING_AGENT" = "0" ]; then
        # Launch a new instance of the agent
        ssh-agent -s -t 14400 &> ~/.ssh/ssh-agent
   fi
   eval `cat ~/.ssh/ssh-agent`
fi

Then from within VSCode set the Remote SSH Option: Remote.SSH Path: C:\\Users\\YourUser\\ssh.bat I just did that and turned off the Windows SSH agent. Hopefully that will help a portion of you.

Edit: If it doesn't work, you may need to set your default WSL with the command: wsl --setdefault ubuntu-20.04 or whatever instance you want to use.

[ahicks92]

Another way to work around this if connecting to local VMs is to have a local-only passwordless key that can connect only to those VMs. This doesn't widen the impact because anyone with access to the laptop or w/e would have access to the VM data anyway.

(not always applicable of course but I don't do dev on remote servers directly)

[drujd]

Optionally, the PowerShell script can also be executed on logoff instead of shutdown by navigating to User Configuration instead of Computer Configuration.

Actually, it ONLY works as a user logoff script, NOT as shutdown script run by SYSTEM. "ssh-add -D" must be run as the same user as ssh-agent.

This doesn't work (well, unless you run ssh-agent as SYSTEM):

4. Navigate to `Local Computer Policy > Computer Configuration > Windows Settings > Scripts (Startup/Shutdown) > Shutdown

[vixie]

Yeah none of that is useful. If I've got to fight Microsoft on the security policy they intend to enforce, I'll just avoid their version of ssh. Sad but not new.

Microsoft, your registry is not safe for my unlocked keys. Please stop insisting that I ignore the evidence of my senses and my own ability to reason. You are a supplier not an occupier.

p vixie

[logicalextreme]

I just wasted over an hour trying to figure out what exactly Microsoft must be doing to cache my passphrase or decrypted key after noticing I was only getting prompts from Pageant (I still use a PPK version for other stuff).

This is mad behaviour. I get that the plaintext key is encrypted with the DPAPI, but nobody familiar with OpenSSH or any SSH agent that I'm aware of would expect this behaviour. You start the agent, you add your key, you decrypt it with your passphrase, and then if the key is removed from the agent or the agent is terminated, that key ceases to exist.

If it's going to store it in the registry like this then I don't see much point in the agent, other than being able to add keys stored outside the default path. For my use case I may as well not use the agent and store my key unencrypted in the default location in my home directory — it's still protected by the DPAPI, so anything more than that is really security through obscurity.

At first I wasn't sure whether I was more surprised by the fact this behaviour exists, the fact that it uses the Registry of all things, or that this issue has been open for almost five years without a fix. However I then realised that the behaviour's even worse than it seems and I can't see anyone having mentioned this above — not only do decrypted keys persist in the registry across reboots and agent restarts, they persist even if you delete the key files from disk and restart the agent. This may not surprise anybody familiar with this issue as the problem is that the in-memory behaviour has simply been changed to use the Registry and thus no reference to the original filepath is stored (as far as I can tell…), but I think it highlights how bonkers this is.

The behaviour of the original OpenSSH agent was modified on purpose making it less secure, and when you consider that this behaviour affects private keys and certificates it lowers the overall security of multiple systems and resources way beyond the machine running the agent. I've been really enjoying using Windows Terminal and the built-in OpenSSH client so I'm going to try out KeeAgent (thanks for the pointer @egfx-notifications!).

However it's been interesting reading the posts above and getting a feel for the difficulties that have been faced in making an "official" WIndows OpenSSH implementation. All I can do is echo others in saying that the behaviour documented in this issue really does heavily violate the principle of least astonishment for many users, and I believe that adding a section explaining the behaviour (how keys are stored, that this storage is permanent, and how to remove them) to this page on the MS docs (or even a new page in the same area documenting the agent and covering basic usage) would go a long way towards mitigating this astonishment. And even save people some time :)

[bitkat]

Shouldn't fixing this get some priority in light of Microsoft's Secure Future Initiative?

[vixie]

Shouldn't fixing this get some priority in light of Microsoft's Secure Future Initiative?

security is in the mind of the beholder. if microsoft thought keeping a copy of my unlocked key in their persistent registry wasn't secure, they would not have put it there. so there's no question that they won't be taking it out, even as an option let alone as the default. my workaround was to uninstall this and install https://www.msys2.org/ instead. (this also comes with a pretty decent terminal emulator.) this uses ssh-agent in the traditional way (similar to pageant) such that when the agent dies, the unlocked key in its memory heap dies with it. just like bsd and linux. i think microsoft will never stop embracing-and-extending.

[bitkat]

i think microsoft will never stop embracing-and-extending.

It is productive nor necessary to discuss and speculate about MS' motivations. To me, this is a simple and rather obvious error that still needs to be corrected. The name "ssh-agent" is widely understood to imply certain essential functionality, but expectations are not met. Either correct that, or rename the program to protect users from unwittingly walking into a trap. The SFI has both "Secure by design" and "Secure by default" as stated security principles so there's little wiggle room here.

So, yes, they may have thought of the current behaviour as secure, which caused the issue. Time to re-think. SFI provides guidance.