Support for Powershell over SSH

[grv87]

Description

I would like to use SSH transport for Windows machines. I couldn't make Test Kitchen work via WinRM. And, since my machines will be used via SSH anyway, I would like to use it for Test Kitchen too.

Kitchen Version

1.20.0

Ruby Version

ruby 2.5.0p0 (2017-12-25 revision 61468) [x64-mingw32]

Platform Version

Host: Windows 7 SP1 Target: Windows Server 2016 Base

Replication Case

In this gist:

• User data file to install OpenSSH and put SSH public key to authorized keys
• .kitchen.yml
• logs

Script in user data file works, tested with AWS. PuTTy is able to connect to just launched machine, and shell is PowerShell.

Kitchen Output

Preparing solo.rb
Invalid parameter in function: (null). File: (null) Line: 0
Expression: (null)
Converge failed on instance <default-Windows>.
------Exception-------
Class: Kitchen::ActionFailed
Message: SSH exited (1) for command: [$env:TEST_KITCHEN = "1"
$dirs = @("$env:TEMP\kitchen\clients", "$env:TEMP\kitchen\cookbooks", "$env:TEMP\kitchen\data", "$env:TEMP\kitchen\data_bags", "$env:TEMP\kitchen\encrypted_data_bag_secret", "$env:TEMP\kitchen\environments", "$env:TEMP\kitchen\roles")
$root_path = "$env:TEMP\kitchen"

Function Delete-AllDirs($dirs) {
  $dirs | ForEach-Object {
    if (Test-Path ($path = Unresolve-Path $_)) { Remove-Item $path -Recurse -Force }
  }
}

Function Unresolve-Path($p) {
  if ($p -eq $null) { return $null }
  else { return $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($p) }
}

Function Make-RootPath($p) {
  $p = Unresolve-Path $p
  if (-Not (Test-Path $p)) { New-Item $p -ItemType directory | Out-Null }
}

Delete-AllDirs $dirs
Make-RootPath $root_path
] 

Full logs see in the aforementioned gist.

[cheeseplus]

The issue right now is the shell_type which as per https://gist.github.com/grv87/4716842855d8934f3434b3548e899107#file-output-kitchen-diagnose-L101 is set to powershell as that's the default anytime the platform is named something like "windows" (case insensitive). Try setting the shell_type: bourne like so:

platforms:
  - name: Windows
    shell_type: bourne

There might be further challenges depending on what tasks you're trying to accomplish and Windows execution contexts/permissions.

[grv87]

The shell on the other side is PowerShell.

[cheeseplus]

You can't SSH into Powershell, that simply isn't a thing kitchen/chef/tooling supports. Either you use WinRM and use Powershell as the shell OR use SSH and a bourne like shell. SSH + PowerShell is not a supported configuration, in part due to a number of assumptions that WinRM and PowerShell go hand in hand - formerly SSH required third party SSH clients that didn't give proper levels of access or execution context so were worse than WinRM.

This might be possible in the future given Microsoft has introduced OpenSSH Server to the Window 10 line but only as of the Fall Creators Update which is neither ubiquitous or supported by most tooling at this time.

What SSH server is the node running? is this something in-built?

[grv87]

Well, I can with PuTTy: But not with Test Kitchen.

See https://github.com/powershell/Win32-OpenSSH/wiki

UPD: OpenSSH server (under Windows) allows to use different shells. I use PowerShell (default), see argument /PathSpecsToProbeForShellEXEString:$Env:ProgramFiles\PowerShell*\Powershell.exe;$Env:SystemRoot\system32\windowspowershell\v1.0\powershell.exe at setup-ssh.xml.

[cheeseplus]

Again the tooling doesn't support that configuration even if it were useful and didn't have a host of side effect problems. This might be an option someday in the future but just because Putty works isn't a good indication or even compelling evidence that kitchen should work in the same manner.

As I've said, this might change but currently remains unsupported for a variety of reasons. If you'd like to endeavor on work to fix that we are happy to review PRs but there isn't a compelling push in the Windows ecosystem to move to Powershell over SSH yet and we're still supporting the long tail of what we actually know works.

[grv87]

I got it.

I believe SSH is more convenient transport than WinRM, and Test Kitchen would greatly benefit having this feature. This could be a problem with OpenSSH itself, since they consider it not production ready. But better be ready when they finally make stable release.

Could this issue stay here opened as feature request / epic?

Are there technical reasons among the variety of reasons why this is unsupported now?

[cheeseplus]

To be clear: we totally agree SSH is objectively superior as a transport, nobody really thinks otherwise but there is historical context that is missing.

Way long ago, the only way to use kitchen with Windows instances was SSH but one had to run a third-party SSH server and use a bash shell and it was very weird but some folks made it work. Over the last 5 years WinRM/libraries got a lot better and the community general started pushing toward that as the preferred transport for Windows (especially with respect to Win 2008 all the way to present). Microsoft itself has shifted stances several times along the way but the core message here is that WinRM became the dominant and preferred transport for several products (Kitchen, Vagrant, Packer, Ansible, etc) despite it's shortcomings - it's notably slow and it's not meant as an interactive transport in the same way SSH works. It also might need to be "elevated" in order to install certain pieces of software due to execution contexts imposed by Windows so it's not always super friendly and definitely completely unlike what folks expect out of SSH.

The only Windows versions that have SSH natively supported are Windows 10 and 2016 but as mentioned earlier, that official support is barely there and still considered beta. On top of that there are open questions around how powershell under SSH works - morevoer CMD is the default at present. So in terms of setting sane defaults and doing this across multiple versions of Windows is incredibly problematic at this time.

Talking to some of our internal Windows pros the feeling is that this is something that will be "the new normal in 5 years. MS is going to be shifting focus from the winrm transport to ssh but it is super early days."

Is this a reasonable new feature? Totally Is anyone close to using this aside from MS Insiders? Not even close

Are there known specific technical reasons? Not with the absolute latest stuff - it's just that nothing has been designed to handle this case as most Windows-centric shops aren't keen to run a beta/non-MS sponsored (in the past) SSH server on their boxes. Only as of December 2017 has there even been official-beta support in non-insider versions of Windows so this very much the bleeding edge of an ecosystem notorious for it's long-tail.

tl;dr - Powershell over SSH looks like the direction things are heading but that future isn't evenly distributed yet

[grv87]

The bleeding edge Windows 10 feature you are talking about is another implementation. From the wiki at the link I mentioned:

On Windows 10, if you've enabled Developer Mode, you probably have another implementation of SSH installed on your machine

Win32-OpenSSH implementation is separately installed software, not a Windows feature. And is not limited to Windows 10/2016. AFAIK it's just the reincarnation of old attempts to port original OpenSSH to Windows.

[cheeseplus]

I'm referring to the native impl of SSH for Windows described here: https://poweruser.blog/enabling-the-hidden-openssh-server-in-windows-10-fall-creators-update-1709-and-why-its-great-51c9d06db8df

If we were to support this, we're going to target the native feature not a third-party/external install that doesn't come baked into images by default. It's not that we can't/won't support the Win32-OpenSSH it's just such a statistically niche set of users nor do we have any customers expressed interested in that capability that it's not worth the effort at present. This is always subject to change but with a native impl on the horizon and most folks being content with WinRM being the standard it's just not going to be a priority anytime soon.

[bender-the-greatest]

What is the current status of this? Microsoft recommends using SSH when going from Linux to Windows' nodes and does not support issues arising when third-party WinRM clients are being used.