Forced shutdown after vagrant up with error even though vm seems to be properly booted

arnegroskurth commented 6 years ago

Vagrant version

Vagrant 2.1.2

Host operating system

Ubuntu 18.04.1 LTS (Bionic) Linux arne-Latitude-E5570 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Guest operating system

hashicorp/precise64

But tested also with debian/jessie64, ubuntu/trusty64 and ubuntu/xenial64 with the same result

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/precise64"
end

Debug output

https://gist.github.com/arnegroskurth/7e01075805184a89742d87951ce37d1c

Expected behavior

vagrant up should result in a running VM

Actual behavior

vagrant up aborts with the error shown below:

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'debian/jessie64' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'debian/jessie64'
    default: URL: https://vagrantcloud.com/debian/jessie64
==> default: Adding box 'debian/jessie64' (v8.11.0) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/debian/boxes/jessie64/versions/8.11.0/providers/virtualbox.box
==> default: Successfully added box 'debian/jessie64' (v8.11.0) for 'virtualbox'!
==> default: Importing base box 'debian/jessie64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'debian/jessie64' is up to date...
==> default: Setting the name of the VM: test_default_1534326370076_42863
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
/usr/lib/ruby/vendor_ruby/net/ssh/authentication/agent.rb:126:in `block in identities': can't modify frozen NilClass (RuntimeError)
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/agent.rb:120:in `times'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/agent.rb:120:in `identities'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/key_manager.rb:104:in `each_identity'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/session.rb:80:in `block in authenticate'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/session.rb:66:in `each'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/session.rb:66:in `authenticate'
    from /usr/lib/ruby/vendor_ruby/net/ssh.rb:241:in `start'
    from /tmp/.mount_vagrang2sYoE/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:415:in `block (2 levels) in connect'
    from /usr/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
    from /tmp/.mount_vagrang2sYoE/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:389:in `block in connect'
    from /tmp/.mount_vagrang2sYoE/usr/gembundle/gems/vagrant-2.1.2/lib/vagrant/util/retryable.rb:17:in `retryable'
    from /tmp/.mount_vagrang2sYoE/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:388:in `connect'
    from /tmp/.mount_vagrang2sYoE/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:84:in `block in wait_for_ready'
    from /usr/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
    from /tmp/.mount_vagrang2sYoE/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:62:in `wait_for_ready'
    from /tmp/.mount_vagrang2sYoE/usr/gembundle/gems/vagrant-2.1.2/lib/vagrant/action/builtin/wait_for_communicator.rb:16:in `block in call'

Steps to reproduce

Setup Ubuntu 18.04.1 host
Download vagrant 2.1.2 from vagrantup.com (64-bit Linux, md5sum 8232cfe40b32f5439a2e39fd15bc4789)
vagrant init hashicorp/precise64
vagrant up

References

Could not find any similar issues but I have some clues that might help:

I can see the VM booting up properly up to the login-prompt in the Virtualbox Manager
I had vagrant running fine under Ubuntu 17.10 on the same machine (but don't know if version from package manager has been used)
Using the vagrant version (2.0.2) from the ubuntu package manager results in the same behaviour
Virtualbox without vagrant is usable without any problems

arnegroskurth commented 6 years ago

Found the problem: Having a ssh key file installed under ~/.ssh results in this error. This should definitely be supported or at least a proper error-message should appear.

briancain commented 6 years ago

Hi @arnegroskurth - do you have a reason why you think having an ssh key under ~/.ssh caused your error? I just tried to reproduce your issue following the steps you outlined and could not:

vagrant@vagrant:~/test$ ls -lah ~/.ssh
total 20K
drwx------ 2 vagrant root    4.0K Aug 15 16:16 .
drwxr-xr-x 9 vagrant vagrant 4.0K Aug 15 16:14 ..
-rw------- 1 vagrant vagrant  389 Aug 15 15:31 authorized_keys
-rw------- 1 vagrant vagrant 1.7K Aug 15 16:16 id_rsa
-rw-r--r-- 1 vagrant vagrant  397 Aug 15 16:16 id_rsa.pub
vagrant@vagrant:~/test$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'hashicorp/precise64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'hashicorp/precise64' is up to date...
==> default: Setting the name of the VM: test_default_1534349826346_85708
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default:
    default: Guest Additions Version: 4.2.0
    default: VirtualBox Version: 5.2
==> default: Mounting shared folders...
    default: /vagrant => /home/vagrant/test
vagrant@vagrant:~/test$ cat Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/precise64"
end
vagrant@vagrant:~/test$ vagrant --version
Vagrant 2.1.2

arnegroskurth commented 6 years ago

Hi @briancain - thanks for your reply! Strange... I can reproduce the error reliably by just renaming the ssh key:

$ ls -lah ~/.ssh
insgesamt 24K
drwx------  2 arne arne 4,0K Aug 15 18:55 .
drwxr-xr-x 33 arne arne 4,0K Aug 15 18:56 ..
-rw-------  1 arne arne 1,7K Aug 15 12:41 id_rsa
-rw-------  1 arne arne  468 Nov  5  2017 id_rsa.pub
-rw-------  1 arne arne 1,5K Nov  5  2017 id_rsa.putty
-rw-r--r--  1 arne arne  222 Aug 15 11:21 known_hosts

$ vagrant init hashicorp/precise64

$ ~/bin/vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'hashicorp/precise64' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'hashicorp/precise64'
    default: URL: https://vagrantcloud.com/hashicorp/precise64
==> default: Adding box 'hashicorp/precise64' (v1.1.0) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/hashicorp/boxes/precise64/versions/1.1.0/providers/virtualbox.box
==> default: Successfully added box 'hashicorp/precise64' (v1.1.0) for 'virtualbox'!
==> default: Importing base box 'hashicorp/precise64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'hashicorp/precise64' is up to date...
==> default: Setting the name of the VM: test_default_1534352302275_5986
Vagrant is currently configured to create VirtualBox synced folders with
the `SharedFoldersEnableSymlinksCreate` option enabled. If the Vagrant
guest is not trusted, you may want to disable this option. For more
information on this option, please refer to the VirtualBox manual:

  https://www.virtualbox.org/manual/ch04.html#sharedfolders

This option can be disabled globally with an environment variable:

  VAGRANT_DISABLE_VBOXSYMLINKCREATE=1

or on a per folder basis within the Vagrantfile:

  config.vm.synced_folder '/host/path', '/guest/path', SharedFoldersEnableSymlinksCreate: false
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
/usr/lib/ruby/vendor_ruby/net/ssh/authentication/agent.rb:126:in `block in identities': can't modify frozen NilClass (RuntimeError)
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/agent.rb:120:in `times'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/agent.rb:120:in `identities'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/key_manager.rb:104:in `each_identity'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/session.rb:80:in `block in authenticate'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/session.rb:66:in `each'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/session.rb:66:in `authenticate'
    from /usr/lib/ruby/vendor_ruby/net/ssh.rb:241:in `start'
    from /tmp/.mount_vagranVEmXA9/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:415:in `block (2 levels) in connect'
    from /usr/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
    from /tmp/.mount_vagranVEmXA9/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:389:in `block in connect'
    from /tmp/.mount_vagranVEmXA9/usr/gembundle/gems/vagrant-2.1.2/lib/vagrant/util/retryable.rb:17:in `retryable'
    from /tmp/.mount_vagranVEmXA9/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:388:in `connect'
    from /tmp/.mount_vagranVEmXA9/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:84:in `block in wait_for_ready'
    from /usr/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
    from /tmp/.mount_vagranVEmXA9/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:62:in `wait_for_ready'
    from /tmp/.mount_vagranVEmXA9/usr/gembundle/gems/vagrant-2.1.2/lib/vagrant/action/builtin/wait_for_communicator.rb:16:in `block in call'

$ mv ~/.ssh/id_rsa ~/.ssh/id_rsa.bak && mv ~/.ssh/id_rsa.pub ~/.ssh/id_rsa.pub.bak

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'hashicorp/precise64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'hashicorp/precise64' is up to date...
==> default: Setting the name of the VM: test_default_1534352414135_67083
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default: 
    default: Guest Additions Version: 4.2.0
    default: VirtualBox Version: 5.2
==> default: Mounting shared folders...
    default: /vagrant => /home/arne/test

$ vagrant halt
==> default: Attempting graceful shutdown of VM...

$ mv ~/.ssh/id_rsa.bak ~/.ssh/id_rsa && mv ~/.ssh/id_rsa.pub.bak ~/.ssh/id_rsa.pub

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'hashicorp/precise64' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
/usr/lib/ruby/vendor_ruby/net/ssh/authentication/agent.rb:126:in `block in identities': can't modify frozen NilClass (RuntimeError)
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/agent.rb:120:in `times'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/agent.rb:120:in `identities'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/key_manager.rb:104:in `each_identity'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/session.rb:80:in `block in authenticate'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/session.rb:66:in `each'
    from /usr/lib/ruby/vendor_ruby/net/ssh/authentication/session.rb:66:in `authenticate'
    from /usr/lib/ruby/vendor_ruby/net/ssh.rb:241:in `start'
    from /tmp/.mount_vagranEFuHtu/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:415:in `block (2 levels) in connect'
    from /usr/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
    from /tmp/.mount_vagranEFuHtu/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:389:in `block in connect'
    from /tmp/.mount_vagranEFuHtu/usr/gembundle/gems/vagrant-2.1.2/lib/vagrant/util/retryable.rb:17:in `retryable'
    from /tmp/.mount_vagranEFuHtu/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:388:in `connect'
    from /tmp/.mount_vagranEFuHtu/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:84:in `block in wait_for_ready'
    from /usr/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /usr/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
    from /tmp/.mount_vagranEFuHtu/usr/gembundle/gems/vagrant-2.1.2/plugins/communicators/ssh/communicator.rb:62:in `wait_for_ready'
    from /tmp/.mount_vagranEFuHtu/usr/gembundle/gems/vagrant-2.1.2/lib/vagrant/action/builtin/wait_for_communicator.rb:16:in `block in call'

briancain commented 6 years ago

Hi @arnegroskurth - how did you install Vagrant? I noticed in your debug gist that your install directory is different from what the usual .deb package installs to. For example when I use the latest deb package:

vagrant@vagrant:~/test$ vagrant up --debug
 INFO global: Vagrant version: 2.1.2
 INFO global: Ruby version: 2.4.4
 INFO global: RubyGems version: 2.6.14.1
 INFO global: VAGRANT_INSTALLER_ENV="1"
 INFO global: VAGRANT_INSTALLER_VERSION="2"
 INFO global: VAGRANT_EXECUTABLE="/opt/vagrant/embedded/gems/2.1.2/gems/vagrant-2.1.2/bin/vagrant"
 INFO global: VAGRANT_INSTALLER_EMBEDDED_DIR="/opt/vagrant/embedded"
 INFO global: VAGRANT_LOG="debug"
...

It looks like you might be using the appimage installer which could be why I couldn't reproduce the issue. This line is from your debug gist....

 INFO global: VAGRANT_APPIMAGE="1"

arnegroskurth commented 6 years ago

@briancain - I just downloaded and executed the 64-bit Linux file. But installing it using the *.deb package does not change the error: https://gist.github.com/arnegroskurth/d815912bca8b861bc1952e4b344c4540

lvets commented 6 years ago

I have the same problem, but the ssh 'trick' doesn't work for me... Error message I get:

...
==> node1: Running 'pre-boot' VM customizations...
==> node1: Booting VM...
==> node1: Waiting for machine to boot. This may take a few minutes...
    node1: SSH address: 127.0.0.1:2222
    node1: SSH username: vagrant
    node1: SSH auth method: private key
==> node1: Forcing shutdown of VM...
==> node1: Destroying VM and associated drives...
/opt/vagrant/embedded/gems/2.1.4/gems/net-ssh-5.0.2/lib/net/ssh/authentication/agent.rb:128:in `block in identities': can't modify frozen NilClass (RuntimeError)
    from /opt/vagrant/embedded/gems/2.1.4/gems/net-ssh-5.0.2/lib/net/ssh/authentication/agent.rb:122:in `times'
    from /opt/vagrant/embedded/gems/2.1.4/gems/net-ssh-5.0.2/lib/net/ssh/authentication/agent.rb:122:in `identities'
    from /opt/vagrant/embedded/gems/2.1.4/gems/net-ssh-5.0.2/lib/net/ssh/authentication/key_manager.rb:104:in `each_identity'
    from /opt/vagrant/embedded/gems/2.1.4/gems/net-ssh-5.0.2/lib/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
    from /opt/vagrant/embedded/gems/2.1.4/gems/net-ssh-5.0.2/lib/net/ssh/authentication/session.rb:85:in `block in authenticate'
    from /opt/vagrant/embedded/gems/2.1.4/gems/net-ssh-5.0.2/lib/net/ssh/authentication/session.rb:71:in `each'
    from /opt/vagrant/embedded/gems/2.1.4/gems/net-ssh-5.0.2/lib/net/ssh/authentication/session.rb:71:in `authenticate'
    from /opt/vagrant/embedded/gems/2.1.4/gems/net-ssh-5.0.2/lib/net/ssh.rb:246:in `start'
    from /opt/vagrant/embedded/gems/2.1.4/gems/vagrant-2.1.4/plugins/communicators/ssh/communicator.rb:415:in `block (2 levels) in connect'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
    from /opt/vagrant/embedded/gems/2.1.4/gems/vagrant-2.1.4/plugins/communicators/ssh/communicator.rb:389:in `block in connect'
    from /opt/vagrant/embedded/gems/2.1.4/gems/vagrant-2.1.4/lib/vagrant/util/retryable.rb:17:in `retryable'
    from /opt/vagrant/embedded/gems/2.1.4/gems/vagrant-2.1.4/plugins/communicators/ssh/communicator.rb:388:in `connect'
    from /opt/vagrant/embedded/gems/2.1.4/gems/vagrant-2.1.4/plugins/communicators/ssh/communicator.rb:84:in `block in wait_for_ready'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:33:in `catch'
    from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
    from /opt/vagrant/embedded/gems/2.1.4/gems/vagrant-2.1.4/plugins/communicators/ssh/communicator.rb:62:in `wait_for_ready'
    from /opt/vagrant/embedded/gems/2.1.4/gems/vagrant-2.1.4/lib/vagrant/action/builtin/wait_for_communicator.rb:16:in `block in call'

I can provide a full debug output as well.

lvets commented 6 years ago

Lol, I will one up @arnegroskurth. My ~/.ssh folder needs to be empty, except for the config and known_hosts file... :smile: vagrant up is now continuing without issues.

briancain commented 6 years ago

Hi @lvets & @arnegroskurth - I'm curious if you still receive this error with Vagrant 2.1.5. Also, have you at all added your public key to your ssh agent? (i.e. see the results of ssh-add -L)

briancain commented 6 years ago

Another thing that would be good to know: what kind of key did you generate for your public/private key pair?

knowaveragejoe commented 6 years ago

I also have this issue with the following:

Ubuntu 18.04LTS Ruby 2.5.1p57 (2018-03-29 revision 63029) Vagrant 2.1.5

Per your suggestion @briancain, my ssh-agent doesn't even appear to be running(Error connecting to agent: Connection refused).

However, before troubleshooting that, I tried moving my other SSH keys out of the ~/.ssh folder per the other comments. I was able to solve the issue by moving all other SSH keys besides id_rsa/id_rsa.pub out of ~/.ssh/ and into a subfolder underneath. The only other contents of the directory are now the config and authorized_hosts files. Updated my config to point the entries to the new key locations and they continue to work fine.

chrisroberts commented 6 years ago

@arnegroskurth @lvets What kind of keys are you using that are failing to load? I've been trying to reproduce the error you are experiencing and have so far been unsuccessful. I've tracked down the source of the error upstream in the net-ssh library, but would like to be able to have a reproducible case that proves fixes an actual problem.

Thanks!

scottw-finao commented 6 years ago

getting the error under ubuntu 18.04 with vagrant 2.2.0

chrisroberts commented 6 years ago

@scottw-finao What type of private keys do you have included in your ssh-agent? I need to be able to reproduce the issue locally so I can create the appropriate PR with tests upstream for this issue. I can't fix it properly until I can reproduce so input back would be very appreciated. Thanks!

scottw-finao commented 6 years ago

I wasn't clear on what keys they were referring to? Are they talking about keys in the .ssh on the vagrant box itself or on the machine you are ssh'ing "from"? I use rsa1 type keys for most of the stuff around here, but since some systems ask for other encryption types, I have one of each for all the 'types' supported by ssh-keygen.

lvets commented 6 years ago

Does this suffice or do you need more information?

$ ssh-add -l
4096 SHA256:<hash> .ssh/<name1> (RSA)
4096 SHA256:<hash> .ssh/<name2>  (RSA)
4096 SHA256:<hash> .ssh/<name3> (RSA)
2048 SHA256:<hash> .ssh/id_rsa (RSA)
$

What else can I provide?

kikitux commented 6 years ago

Sample to reproduce:

I copy 3 old key in /home/user/.ssh, but included in ssh-agent only id_rsa.

nshalnev@nshalnev:~$ cat .ssh/id_rsa 
-----BEGIN RSA PRIVATE KEY----- 
Proc-Type: 4,ENCRYPTED 
DEK-Info: DES-EDE3-CBC,4895574DB45CF982

-----END RSA PRIVATE KEY-----

It's key genereted on putty in Windows.

After i cut 2 other key out .ssh, vagrant started

davemcphee commented 6 years ago

Exact same issue here; can confirm that running vagrant up as root (which has an empty ~/.ssh) dir works fine.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:    18.04
Codename:   bionic

$ vagrant --version
Vagrant 2.2.0

/opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh/authentication/agent.rb:128:in `block in identities': can't modify frozen NilClass (RuntimeError)
    from /opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh/authentication/agent.rb:122:in `times'
    from /opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh/authentication/agent.rb:122:in `identities'
    from /opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh/authentication/key_manager.rb:104:in `each_identity'
    from /opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
    from /opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh/authentication/session.rb:85:in `block in authenticate'
    from /opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh/authentication/session.rb:71:in `each'
    from /opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh/authentication/session.rb:71:in `authenticate'
    from /opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh.rb:246:in `start'

I assume I have a file in my ~/.ssh folder that's causing this, but visual check of key files shows nothing unusual. --debug shows a bunch of successful ssh operations but nothing related to a specific file. Any way to narrow down what it might be? Some kind of for each ~/.ssh/*; do /opt/vagrant/embedded/gems/2.2.0/gems/net-ssh-5.0.2/lib/net/ssh.rb blah command I could run?

Edit: just saw your latest comment RE which type of key - I guess I'll start copying each file from errorring user's ~/.ssh to working root user's ~/.ssh one by one until it breaks; I'll update in a few days at the latest if I can make it happen.

chrisroberts commented 6 years ago

Hi everyone. Thanks for the input everyone has provided on this issue. I was finally able to reproduce the issue locally, and here is what I found:

The TL;DR of it is this: The origin of the issue is gnome-keyring. The fixed version of gnome-keyring is not available in the ubuntu repos. A public key in the .ssh directory is an openssh v1 public key and it is causing the problem. Remove that public key and the error should not occur.

The long version is this: The commonality of all the posters has been ubuntu 18.04, and items within their user .ssh directory being automatically added to the ssh-agent. This led to the fact that it was gnome-keyring-daemon that was importing keys and providing the ssh-agent functionality. With this I spun up a ubuntu 18.04 system, installed gnome-keyring and started up the gnome-keyring-daemon. After generating a handful of keys of varying types and sizes I was unable to reproduce the error. So I turned to the gnome-keyring source to see what changes have been made recently and I found this commit:

https://gitlab.gnome.org/GNOME/gnome-keyring/commit/65e20169b8dfc935c907ca65e54c1c6e2a365388

After generating a new key with the openssh v1 format this error reproduced itself. Removing the v1 public key allowed it to operate successfully. So identifying any v1 public keys in your .ssh directory and removing them should resolve the issue for now. I'm going to be submitting a fix to net-ssh so it can gracefully handle this kind of error without resulting in a fatal error. And hopefully the latest gnome-keyring will becoming available soon which will also prevent this from happening.

Since all of this lies upstream (gnome-keyring and net-ssh), I'm going to be closing this up now.

Cheers!

chrisroberts commented 6 years ago

Fix for net-ssh library submitted here: net-ssh/net-ssh#640

ghost commented 5 years ago

==> default: Vagrant has detected a configuration issue which exposes a ==> default: vulnerability with the installed version of VirtualBox. The ==> default: current guest is configured to use an E1000 NIC type for a ==> default: network adapter which is vulnerable in this version of VirtualBox. ==> default: Ensure the guest is trusted to use this configuration or update ==> default: the NIC type using one of the methods below: ==> default: ==> default: https://www.vagrantup.com/docs/virtualbox/configuration.html#default-nic-type ==> default: https://www.vagrantup.com/docs/virtualbox/networking.html#virtualbox-nic-type ==> default: Clearing any previously set network interfaces... ==> default: Preparing network interfaces based on configuration... default: Adapter 1: nat ==> default: Forwarding ports... default: 22 (guest) => 2200 (host) (adapter 1) ==> default: Running 'pre-boot' VM customizations... ==> default: Booting VM... ==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 127.0.0.1:2200 default: SSH username: vagrant default: SSH auth method: private key default: default: Vagrant insecure key detected. Vagrant will automatically replace default: this with a newly generated keypair for better security. default: default: Inserting generated public key within guest... ==> default: Forcing shutdown of VM... ==> default: Destroying VM and associated drives... /opt/vagrant/embedded/gems/2.2.2/gems/vagrant-2.2.2/plugins/hosts/linux/cap/ssh.rb:11:in chmod': Operation not permitted @ chmod_internal - /mnt/f/vagrant/.vagrant/machines/default/virtualbox/private_key (Errno::EPERM) from /opt/vagrant/embedded/gems/2.2.2/gems/vagrant-2.2.2/plugins/hosts/linux/cap/ssh.rb:11:inchmod' from /opt/vagrant/embedded/gems/2.2.2/gems/vagrant-2.2.2/plugins/hosts/linux/cap/ssh.rb:11:in set_ssh_key_permissions' from /opt/vagrant/embedded/gems/2.2.2/gems/vagrant-2.2.2/lib/vagrant/capability_host.rb:111:incall' from /opt/vagrant/embedded/gems/2.2.2/gems/vagrant-2.2.2/lib/vagrant/capability_host.rb:111:in capability' from /opt/vagrant/embedded/gems/2.2.2/gems/vagrant-2.2.2/plugins/communicators/ssh/communicator.rb:200:inready?' from /opt/vagrant/embedded/gems/2.2.2/gems/vagrant-2.2.2/plugins/communicators/ssh/communicator.rb:87:in block in wait_for_ready' from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:93:inblock in timeout' from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:33:in block in catch' from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:33:incatch' from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:33:in catch' from /opt/vagrant/embedded/lib/ruby/2.4.0/timeout.rb:108:intimeout' from /opt/vagrant/embedded/gems/2.2.2/gems/vagrant-2.2.2/plugins/communicators/ssh/communicator.rb:62:in wait_for_ready' from /opt/vagrant/embedded/gems/2.2.2/gems/vagrant-2.2.2/lib/vagrant/action/builtin/wait_for_communicator.rb:16:inblock in call'

scottw-finao commented 5 years ago

removing v1 keys of which directory? the machine running vagrant or the machine running 'in' vagrant?

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / vagrant