new ssh config directive "include" breaks "vagrant ssh"

[fourjay]

Vagrant version

Vagrant 2.2.3

Host operating system

Opensuse Leap 15

Vagrantfile

Vagrant.configure(2) do |config|
  # config.vm.box = "opensuse/openSUSE-42.3-x86_64"
   config.vm.box = "opensuse/openSUSE-15.0-x86_64"
  config.ssh.insert_key = false
  {
      'vagrant' => '10.0.93.2',
  }.each do |short_name, ip|
    config.vm.define short_name do |host|
      host.vm.network 'private_network', ip: ip
      host.vm.hostname = "#{short_name}.myapp.dev"
    end
  end
end

~/.ssh/config

# this speeds up parallel ssh
ControlMaster auto
ControlPath /tmp/ssh_mux_%h_%p_%r
# minor ssh tweaks
include ~/.dotfiles/.ssh/include/personal

Expected behavior

vagrant ssh should connect to vagrant box

Actual behavior

/home/XXX/.ssh/config: terminating, 5 bad configuration options

• Can ssh in via dumping ssh config and loading it Via -F
• can include the dumped config in ssh config and ssh via ssh vagrant can workaround by adding config.ssh.config = "/dev/null"

Steps to reproduce

1. add an include directive in .ssh/config

[chrisroberts]

Hi there,

Would you please provide a gist of the debug output from running: vagrant ssh --debug

Thanks!

[fourjay]

Here is a gist of the debug output (sorry for the delay) https://gist.github.com/fourjay/8636b3ebee51da7493ca3c52dbe5c630

[blairlyrical]

vagrant.debug.log

I"m having the same issue. I've attached the debug output.

For this test, my ~/.ssh/config file only has the include: include conf.d/*

My conf.d has a file that only has this:

Host *
  StrictHostKeyChecking no
  UserKnownHostsFile=/dev/null

I tried to eliminate as much fluff as possible to narrow down the issue.

Thanks in advance!

[fcoelho]

Also hitting the same issue using the AppImage Vagrant 2.2.4, in my case a ProxyJump option is causing it to fail. Removing it makes vagrant work as expected. My ~/.ssh/config file is literally the following:

Host dummy
    ProxyJump user@nowhere

[blairlyrical]

Also hitting the same issue using the AppImage Vagrant 2.2.4, in my case a ProxyJump option is causing it to fail. Removing it makes vagrant work as expected. My ~/.ssh/config file is literally the following:

Host dummy
  ProxyJump user@nowhere

I had exactly the same thing in mine. I set this up to work around it:

Host dev-acme-*
  Hostname %h
  ProxyCommand ssh -W %h:%p jump.acme.corp
  IdentityFile ~/.ssh/acme-key
  User acme

That will get you with a working vagrant and a working proxy jump. Lame, but functioning.

Hope it helps.

[hoshsadiq]

Same issue here. Version 2.2.4

[hoshsadiq]

To add to @fourjay's workaround, that needs to be only run when using vagrant ssh, as for me all other commands refused to work with that config option.

  if ARGV[0] == 'ssh'
    config.ssh.config = "/dev/null"
  end

[dragetd]

This workaround does not work for me when using provision

SSH:
* `config` file must exist: /dev/null

Hmm =/

[hoshsadiq]

@dragetd are you using windows? If so, instead of /dev/null try creating an empty file somewhere and pointing to that.

[dragetd]

No, on gentoo/Linux. Also tried to create an empty file and point the config there with no success.

I joined my SSH Config with cat .ssh/config.d/* > .ssh/config for now, and then got even another bad configuration options for AddKeysToAgent yes. I was trying to find the responsible code in net-ssh/net-ssh but failed to find the string 'bad configuration option'… also it seems like there is already support for the 'Include' statement, as I mentioned in the other issue.

Basically I was even unable to figure out how vagrant does SSH xD

[dovry]

ProxyJump user@jumphost:1234

^ This was breaking every box I tried

➜ vagrant ssh
/home/user/.ssh/config: terminating, 1 bad configuration options

˅ Replacement/workaround

Proxycommand ssh user@jumphost -p 1234 nc %h %p

[vrubiolo]

I am hitting the same issue with JumpProxy using Vagrant 2.2.6 (from vagrantup.com) on Fedora 30. Commenting out the JumpProxy directive allows Vagrant to proceed.

Thanks to @Dovry and @blairlyrical for the workaround, I will give this one a try.

I suspect this might be because Vagrant comes with an older version of ssh. I can see the following in the debug log:

 INFO ssh: Invoking SSH: /tmp/.mount_vagranWdxEzX/usr/bin/ssh ["vagrant@127.0.0.1", "-p", "2222", "-o", "LogLevel=FATAL", "-o", "Compression=yes", "-o", "DSAAuthentication=yes", "-o", "IdentitiesOnly=yes", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "-i", "/home/vincent/Documents/Dev/vagrant/.vagrant/machines/default/virtualbox/private_key"]
/home/vincent/.ssh/config: terminating, 1 bad configuration options

The /tmp/.mount_vagranWdxEzX/usr/bin/ssh part is what leads me to think the SSH binary comes shipped with Vagrant itself.

Edit: this is indeed the case, Vagrant seems to ship with OpenSSH v6.6.1 :

After mounting the AppImage, I cannot run the ssh binary directly because of library issues but we can extract the strings from the binary. The first match is the one from the ssh -V output:

$ ./vagrant --appimage-mount &
/tmp/.mount_vagranQbEpxW
$ cd /tmp/.mount_vagranQbEpxW
$ strings usr/bin/ssh | grep OpenSSH_ | head -n1
OpenSSH_6.6.1

The JumpHost directive was introduced w/ OpenSSH 7.3 back in August 2016.

As an aside, it might be good to upgrade the binary, this is more than 3 years old for a security-oriented product like SSH...

Edit2: I confirm @blairlyrical workaround using ProxyCommand ssh -W %h:%p works fine. Thanks!

Edit3: it also seems the doc at https://www.vagrantup.com/docs/cli/ssh.html#ssh-client-usage does not match what is happening: instead of using the host machine ssh client (as the doc says), Vagrant uses its internal one

[dragetd]

Okay, this explains why I failed to find anything about a ruby-based implementation of SSH in vagrant. :-P

And yes, upgrading to a more recent SSH version would solve a bunch of issues! 7.3 is also the version that brought the include statement.

Ubuntu current LTS 18.04 comes with 7.6 and the latest version is 8.1(!). The bundled 6.6.1 also comes with a nice CVE: https://www.openssh.com/txt/release-7.1p2

In the context of vagrant not a major threat, but still a thing to be aware of. I highly recommend upgrading to at least 7.6 since this is maintained by Canonical till 2023. Or directly 8.1.

[vrubiolo]

Hi @dragetd, thanks for the additional information! Security-wise, I was thinking about vagrant connect --ssh or vagrant connect which expose ports of your machine to the outside world (agreed for dev purpose in the normal workflow).

[dragetd]

I can see the ssh process version v6.6.1 when using vagrant ssh. But I am just completely unable to figure out how this works in the source. https://github.com/hashicorp/vagrant/blob/master/plugins/communicators/ssh/plugin.rb#L10 implies that ruby net-ssh is used. Is this the import for net-ssh? https://github.com/hashicorp/vagrant/blob/b1d8b952bb4da7e18782f6e3422cfe5e99014690/plugins/communicators/ssh/communicator.rb#L9

net-ssh has a similar open issue at https://github.com/net-ssh/net-ssh/issues/650 eventho the source implies that it does actually parse the command (see other issue).

But the running SSH binary when calling vagrant ssh is an elf binary, not a ruby module. How does a ruby ssh implementation spawn an outdated ssh binary?

The net-ssh version used by vagrant is 5.2, which is not the latest one, tho the newer versions do not contain any fixes in this direction. https://github.com/hashicorp/vagrant/blob/master/vagrant.gemspec#L26

I got it to work with defining

  if ARGV[0] == 'ssh'
    config.ssh.config = "/dev/null"
  end

in my config, no clue why it did not work some weeks ago. Still, the whole thing is a bit frustrating. =/

[dragetd]

I got it working was eventually only related to using vagrant ssh

Now that I came back to it while not using my config-mashing-hack anymore, I am unable to run vagrant up no matter where I put that workaround.

sigh

[nilium]

Seeing this as well regardless of provider used. Any chance of getting an updated OpenSSH in the Vagrant AppImage? If it's going to read a user's SSH configs, it seems important that it be fairly recent, not a version that's so far behind that it breaks on common config directives.

[vrubiolo]

Indeed, an actual ssh ELF binary is invoked in the AppImage (as I saw above).

I am unsure however about the relationship between this and what @dragetd saw above though (which uses net-ssh and not a native binary).

[vrubiolo]

Hit the bug again today (Fedora 32, Vagrant 2.29) as I added an Include directive to my ssh config to split it into manageable chunks.

The workaround from @fourjay does not seem to work anymore as Vagrant appears to check for the existence of the SSH config file:

$ grep null Vagrantfile 
  config.ssh.config = "/dev/null"
$ vagrant reload
There are errors in the configuration of this machine. Please fix
the following errors and try again:
SSH:
* `config` file must exist: /dev/null

Using an almost empty file does the trick though:

$ cat ~/tmp/fake_ssh_config 
# Fake SSH config file for Vagrant issue #10601
$ grep ssh Vagrantfile 
  config.ssh.config = "/home/vrubiolo/tmp/fake_ssh_config"

@chrisroberts : is there any interested in getting this fixed in mainline Vagrant? I am surprised no more people from the core team are hitting this (or maybe they are using another workaround) ?

[unfulvio]

Same issue here after updating to Fedora 32 past week

• update:

https://github.com/hashicorp/vagrant/issues/10601#issuecomment-503962378

this solution worked for me

[vrubiolo]

@unfulvio : thanks for the feedback. Which version of Vagrant are you using? I have found that mine does now allow the /dev/null workaround anymore (cf :arrow_up: )

[unfulvio]

@vrubiolo v2.2.9 - I am using it with this project: https://github.com/Varying-Vagrant-Vagrants/VVV

[vrubiolo]

@unfulvio : thanks for the feedback. This is interesting. I am also using 2.2.9 and cannot pass through the /dev/null error (cf my post above. Don't you get the error I have?

[unfulvio]

@vrubiolo frankly no, I don't know why - I have used vagrant ssh after vagrant up - if the empty file does the trick for you I guess you may just use that until the bug is fixed. I have added https://github.com/hashicorp/vagrant/issues/10601#issuecomment-503962378 snippet at the beginning of my Customfile

[reini-1]

Try

# vagrant build in ssh version:
$ vagrant ssh -- -V
OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13, OpenSSL 1.0.1f 6 Jan 2014

# my workaround:
$ vagrant ssh -- -F /dev/null

see also my issue #11709

[vrubiolo]

@reini-1 : thanks for the additional info, esp the clever use of vagrant ssh to get the embedded SSH client version!

[zioalex]

I had the same problem with vagrant 2.2.9. Easily worked around creating an empty file in the Vagrant folder like empty_ssh_config and then used config.ssh.config = "empty_ssh_config" in the Vagrantfile. Everything works fine, provision and reload included.

[DnR-iData]

Same here with latest version (2.2.9 @ today).

11788 is in relation ? :D

[kaosmaja]

11788 is in relation ? :D

It's in relation. Thanks for the mention and thanks zioalex for providing an easier workaround than I came up with.

Still, it does not remove the need for updating the ssh version Vagrant uses.

[RafalSkolasinski]

Just wanted to report that having ProxyJump breaks vagrant ssh for me too.

[jserras]

Greetings,

Any expectations regarding this issue? seems to be some generic issue parsing the users .ssh/config, should it be even parsing it? At least with Include and ProxyJump statements are a problem, or is it just an issue of updating the parser with the current options of ssh? Using Vagrant 2.2.10 and OpenSSH 7.4p1, not even the most recent OpenSSH

[chrisroberts]

Hi everyone,

This issue affects the appimage version of Vagrant and will be resolved in the next release. This PR hashicorp/vagrant-installers#181 strips out a bunch of extras that were included to make it behave much like the system packages which provide dependency applications and libraries. The result is the appimage version will not attempt to use an internal version of ssh (which in this case is an older version that does not support newer configuration options). We're wrapping up the last items for the next release so I'll leave this issue open until the release is published and available.

Cheers!

[GyurkanM]

This issue is still present on RHEL 8.8, OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021, vagrant version 2.4,0. This is the line that causes issues - Include /etc/ssh/ssh_config.d/*.conf

[dragetd]

@GyurkanM Which version of vagrant and disto-package or AppImage?

For me, not using the AppImage version did solve an unrelated issue as well, and currently (vagrant 2.3.6) it works for my SSH config.

I think it was resolved with the next release as @chrisroberts said. Maybe if you got a new problem, open a new issue and we can close this one?

[GyurkanM]

@GyurkanM Which version of vagrant and disto-package or AppImage?

For me, not using the AppImage version did solve an unrelated issue as well, and currently (vagrant 2.3.6) it works for my SSH config.

I think it was resolved with the next release as @chrisroberts said. Maybe if you got a new problem, open a new issue and we can close this one?

Version 2.4.0 as said above and installed with dnf