`vagrant ssh` fails with `Permission denied (publickey)` error on Windows 10

[brookinc]

Using the script on a Windows 10 machine, the setup, boot, and provisioning steps succeed, but running vagrant ssh fails:

D:\Projects\swift-linux-vagrant>vagrant ssh
vagrant@127.0.0.1: Permission denied (publickey).

[brookinc]

It looks like this is an issue with the Windows 10 ssh binary (https://github.com/hashicorp/vagrant/issues/9433), and a workaround will be possible as of Vagrant 2.0.4 by setting VAGRANT_PREFER_SYSTEM_BIN=0 (https://github.com/hashicorp/vagrant/pull/9503). Alternately, running through a shell that doesn't use the Windows 10 ssh binary (for example, cmder) should also work (with any Vagrant version, including the current ones).

[brookinc]

VAGRANT_PREFER_SYSTEM_BIN=0 is now supported in the current Vagrant release, and Vagrantboot.sh has been updated to set that environment variable automatically: https://github.com/brookinc/swift-linux-vagrant/commit/4a9c022c819e6059919f553d2f35ad3d328d2410

[kvietmeier]

This doesn't seem to be working. I just got updated to Build 1709 and installed the OpenSSH Beta. I set "VAGRANT_PREFER_SYSTEM_BIN=0" in both a local Vagrantfile and the global Vagrantfile. Trying to use "vasgrant ssh --debug" reveals that the Windows binary is still being used:

 INFO ssh: Invoking SSH: C:\windows\System32\OpenSSH\/ssh.EXE ["vagrant@127.0.0.1", "-p", "2222", "-o", "LogLevel=FATAL", "-o", "Compression=yes", "-o", "DSAAuthentication=yes", "-o", "IdentitiesOnly=yes", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "-o", "IdentityFile=\"C:/Users/ksvietme/Documents/Projects/vagrant/virtualbox/simplevm/.vagrant/machines/default/virtualbox/private_key\""]
DEBUG safe_exec: Converting command and arguments to common UTF-8 encoding for exec.
DEBUG safe_exec: Command: `"C:\\windows\\System32\\OpenSSH\\/ssh.EXE"` Args: `["vagrant@127.0.0.1", "-p", "2222", "-o", "LogLevel=FATAL", "-o", "Compression=yes", "-o", "DSAAuthentication=yes", "-o", "IdentitiesOnly=yes", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "-o", "IdentityFile=\"C:/Users/ksvietme/Documents/Projects/vagrant/virtualbox/simplevm/.vagrant/machines/default/virtualbox/private_key\""]`
DEBUG safe_exec: Converted - Command: `"C:\\windows\\System32\\OpenSSH\\/ssh.EXE"` Args: `["vagrant@127.0.0.1", "-p", "2222", "-o", "LogLevel=FATAL", "-o", "Compression=yes", "-o", "DSAAuthentication=yes", "-o", "IdentitiesOnly=yes", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "-o", "IdentityFile=\"C:/Users/ksvietme/Documents/Projects/vagrant/virtualbox/simplevm/.vagrant/machines/default/virtualbox/private_key\""]`
vagrant@127.0.0.1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
PS C:\Users\ksvietme\Documents\Projects\vagrant\virtualbox\simplevm>

[brookinc]

Hmm, interesting! Are you using this repo's specific vagrant file (ie. a swift-linux-vagrant repo clone), or is this just something you're seeing with your vagrant setup generally?

For comparison, on my system (Windows 10 v1803, Vagrant 2.1.2, Virtual Box 5.2.18), in a swift-linux-vagrant repo clone, this is (the tail of) the output I get from vagrant ssh --debug:

DEBUG virtualbox_5_2:   - [1, "ssh", 2222, 22, "127.0.0.1"]
 INFO subprocess: Starting process: ["C:\\WINDOWS\\System32\\OpenSSH\\/ssh.EXE"]
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stderr: usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
           [-D [bind_address:]port] [-E log_file] [-e escape_char]
           [-F configfile] [-I pkcs11] [-i identity_file]
           [-J [user@]host[:port]] [-L address] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-p port] [-Q query_option] [-R address]
           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
           destination [command]
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 255
 INFO ssh: Invoking SSH: C:\WINDOWS\System32\OpenSSH\/ssh.EXE ["vagrant@127.0.0.1", "-p", "2222", "-o", "LogLevel=FATAL", "-o", "Compression=yes", "-o", "DSAAuthentication=yes", "-o", "IdentitiesOnly=yes", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "-o", "IdentityFile=\"C:/Users/Brook/Desktop/Data/Projects/swift-linux-vagrant/.vagrant/machines/default/virtualbox/private_key\""]
DEBUG safe_exec: Converting command and arguments to common UTF-8 encoding for exec.
DEBUG safe_exec: Command: `"C:\\WINDOWS\\System32\\OpenSSH\\/ssh.EXE"` Args: `["vagrant@127.0.0.1", "-p", "2222", "-o", "LogLevel=FATAL", "-o", "Compression=yes", "-o", "DSAAuthentication=yes", "-o", "IdentitiesOnly=yes", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "-o", "IdentityFile=\"C:/Users/Brook/Desktop/Data/Projects/swift-linux-vagrant/.vagrant/machines/default/virtualbox/private_key\""]`
DEBUG safe_exec: Converted - Command: `"C:\\WINDOWS\\System32\\OpenSSH\\/ssh.EXE"` Args: `["vagrant@127.0.0.1", "-p", "2222", "-o", "LogLevel=FATAL", "-o", "Compression=yes", "-o", "DSAAuthentication=yes", "-o", "IdentitiesOnly=yes", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "-o", "IdentityFile=\"C:/Users/Brook/Desktop/Data/Projects/swift-linux-vagrant/.vagrant/machines/default/virtualbox/private_key\""]`
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-134-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

vagrant@ubuntu-xenial:~$

So it looks like essentially the same path that your system is calling, but for some reason it works for me and not for you.

My system is a more or less "vanilla" Windows install -- ie. I haven't installed an OpenSSH beta or anything else along those lines -- so I guess there are a couple of questions here: 1) is it expected that Vagrant would be using OpenSSH when VAGRANT_PREFER_SYSTEM_BIN=0 is set? (That seems wrong...) 2) why does OpenSSH work for vagrant access on some systems (ie. mine) but not on others (ie. yours)?

Unfortunately, the best suggestions I can make are: try updating to the latest release versions of each piece of software involved to see if you still get the same results; see if you can find reports of others experiencing a similar issue; and get in touch with vagrant support to see if they have any ideas...

[brookinc]

One other footnote is that, as might be expected, when I set VAGRANT_PREFER_SYSTEM_BIN to 1 in Vagrantboot.sh, destroyed and re-inited my vagrant box, and logged in again, I saw the same output as above (the system OpenSSH was used, but I was able to successfully log in).

I'm not sure if that means that something has changed on the Windows side or on the vagrant side (or both) since I first encountered this issue, but it does seem like something's different...

[kvietmeier]

@brookinc You are descruibing exactly the behaviour that is a bug. When VAGRANT_PREFER_SYSTEM_BIN is set to 0, it should ignore the builtin SSH client and use the embedded SSH that ships with Vagrant. So you should see different output depending on whether it is "1" or "0", it is a toggle, true/false. Right now it is 1 = true, regardless of the value set. I proveds this to myself by ermoving any SSH clients from my instance and sure enough, Vagrant used it's own SSH client, despite the value of VAGRANT_PREFER_SYSTEM_BIN.

The reason yours is working is because you are using 1803, the April edition, I am using 1709, Fall Creators. The beta in 1709 only supports ed25519 keys and Vagrant uses RSA keys, so it breaks. It looks like 1803 is using RSA keys by default and it works - there were many complaints from end users about this. Enabling ed25519 keys on all of your servers is a non-rtivial excercise in most environments. I remmoved the built-in beta and went back to the opensshe port installed with choclatey which uses RSA keys.

But it looks like I posted this issue in the wrong place - I apologize for that. This isn't a Swift Linux issue.