search

hashicorp / vagrant

Vagrant is a tool for building and distributing development environments.
https://www.vagrantup.com
Other
26.24k stars 4.44k forks source link

Port forward issues on macOS Big Sur with Vagrant VMware provider #12045

Open michalmiddleton opened 3 years ago

michalmiddleton commented 3 years ago

Vagrant version

Vagrant 2.2.13 vagrant-vmware-desktop (2.1.5, global) vagrant-vmware-utility 1.0.16 VMware Fusion Pro 12

Host operating system

macOS 12.0.1 Big Sur

Guest operating system

Linux (centos7) or Windows (2012r2)

Vagrantfile

Vagrant.configure(2) do |config|
    Vagrant.require_version ">= 2.1.5"
    config.vm.box = "tmcsvc/centos-7"

    config.vm.provider :vmware_desktop do |vmware|
        vmware.vmx['displayname'] = 'Linux-CentOS7-test_linux'
        vmware.linked_clone = true
        vmware.gui = false
        vmware.vmx["memsize"] = "1024"
        vmware.vmx["numvcpus"] = "2"
        vmware.vmx["ethernet0.pcislotnumber"] = "33"
        vmware.whitelist_verified = true
    end
end

Debug output

Regular vagrant up

Bringing machine 'default' up with 'vmware_desktop' provider...
==> default: Cloning VMware VM: 'tmcsvc/centos-7'. This can take some time...
==> default: Checking if box 'tmcsvc/centos-7' version '7.8.1' is up to date...
==> default: Verifying vmnet devices are healthy...
==> default: Preparing network adapters...
==> default: Starting the VMware VM...
==> default: Waiting for the VM to receive an address...
==> default: Forwarding ports...
    default: -- 22 => 2222
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
...

netstat shows no open port 2222:


tcp6       0      0  *.53                   *.*                    LISTEN      131072 131072    226      0 0x0100 0x00000006
tcp4       0      0  *.49682                *.*                    LISTEN      131072 131072    226      0 0x0100 0x00000006
tcp4       0      0  127.0.0.1.49227        *.*                    LISTEN      131072 131072   8366      0 0x0180 0x00000026
tcp4       0      0  127.0.0.1.8698         *.*                    LISTEN      131072 131072   6393      0 0x0100 0x00000026
tcp4       0      0  127.0.0.1.53           *.*                    LISTEN      131072 131072   5123      0 0x0180 0x0000000e
tcp4       0      0  127.0.0.1.44444        *.*                    LISTEN      131072 131072   5111      0 0x0080 0x00000006
tcp4       0      0  127.0.0.1.9922         *.*                    LISTEN      131072 131072    382      0 0x0180 0x00000026
tcp4       0      0  *.59866                *.*                    LISTEN      131072 131072    507      0 0x0100 0x00000006
tcp6       0      0  *.49187                *.*                    LISTEN      131072 131072    432      0 0x0100 0x00000006
tcp4       0      0  *.49187                *.*                    LISTEN      131072 131072    432      0 0x0100 0x00000006
tcp46      0      0  *.18102                *.*                    LISTEN      131072 131072     79      0 0x0100 0x00000006
tcp6       0      0  *.61500                *.*                    LISTEN      131072 131072    245      0 0x0180 0x00000006
tcp4       0      0  *.61500                *.*                    LISTEN      131072 131072    245      0 0x0180 0x00000006
tcp4       0      0  127.0.0.1.8021         *.*                    LISTEN      131072 131072      1      0 0x0180 0x00000006
tcp6       0      0  ::1.8021               *.*                    LISTEN      131072 131072      1      0 0x0180 0x00000006```

Full debug: https://gist.github.com/michalmiddleton/8441bef9c65abb697d38005207317052

### Expected behavior

connect to the vm

### Actual behavior

stuck on waiting to connect. Happens for both windows and linux boxes

### Steps to reproduce

1. vagrant up
2.
3.

### References
none
fhenri commented 3 years ago

I have same issue after upgrade

host: macOS bigsur vm: centos vagrant version: latest (2.2.13) vagrant vmware: 2.1.5 vagrant utility : 1.0.16

did you check your file /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf in [incomingtcp] section ? do you have an entry created for the VM ?

GameGamer43 commented 3 years ago

@fhenri I'm having the same issue and have confirmed that the VM is in /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf

As a temporary workaround you can use SSH to forward the port using something like the following command:

ssh -L [ssh port vagrant is mapping to 22]:[vm ip address]:22 vagrant@[vm ip address]

michalmiddleton commented 3 years ago

Same here, the [incomingtcp] entry is in place. So is this actually a Fusion issue? I tried explicitly listing Fusion in my Firewall and allowed Full Disk access (because that was the only place that listed Fusion) but no luck.

michalmiddleton commented 3 years ago

It's listed as a Known Issue with the v12 Fusion release: https://docs.vmware.com/en/VMware-Fusion/12/rn/VMware-Fusion-12-Release-Notes.html (╯°□°)╯︵ ┻━┻

chrisroberts commented 3 years ago

Hi there,

As noted above (https://github.com/hashicorp/vagrant/issues/12045#issuecomment-726872246), port forwarding to the loopback device is no longer functional in Fusion 12 on macOS Big Sur. This is a result of the move away from kexts and to using the vmnet framework provided by macOS. Since Vagrant connects to the guest via the port forward on the loopback, and the port forward isn't actually functional, Vagrant can never establish a connection. We worked for quite a bit to see if we could provide an addon solution to get the port forwarding to work until it's been resolved within Fusion itself, but it'll be a lower level fix that's required.

A workaround for this is to enable connecting to the guest via the assigned IP address directly using ssh_info_public:

Vagrant.configure("2") do |config|
  config.vm.provider :vmware_desktop do |v|
    v.ssh_info_public = true
  end
end

You can also automatically apply this globally to Vagrant projects by adding this to a Vagrantfile in the VAGRANT_HOME directory (~/.vagrant.d/Vagrantfile).

michalmiddleton commented 3 years ago

@chrisroberts Thank you! Alvaro (via Hashi support ticket) just replied with the same config change :) This works!!!

I opened a ticked with VMware to see what they say about it as well.

fhenri commented 3 years ago

yep working for me too

ghsolutions commented 3 years ago

works for me too

rbmcgowan commented 3 years ago

A workaround for this is to enable connecting to the guest via the assigned IP address directly using ssh_info_public:

Vagrant.configure("2") do |config|
  config.vm.provider :vmware_desktop do |v|
    v.ssh_info_public = true
  end
end

@chrisroberts, vagrant novice here, do we just paste that workaround into the Vagrantfile ? because I tried that without any success?

rbmcgowan commented 3 years ago

adding the following to the bottom of my Vagrantfile worked for me (using vmware_fusion rather than vmware_desktop).

Vagrant.configure("2") do |config|
  config.vm.provider : vmware_fusion do |v|
    v.ssh_info_public = true
  end
end
lukepolo commented 3 years ago

Seems to be the same problem with virtual box https://github.com/hashicorp/vagrant/issues/12047 , but not sure what the workaround is for that

sengfai commented 3 years ago

Hi everyone I solved it:

  1. Update your virtualbox
  2. Update your vagrant
  3. Go to System Preference then privacy& security and Allow the permission in general tap.
4levels commented 3 years ago

The proposed solution of adding the ssh_info_public option throws an error when using virtualbox provider:

There are errors in the configuration of this machine. Please fix
the following errors and try again:
VirtualBox Provider:
* The following settings shouldn't exist: ssh_info_public
chrisroberts commented 3 years ago

Please note that this issue is specifically about the Vagrant VMware Desktop provider. This issue does not affect Virtualbox on macOS Big Sur.

4levels commented 3 years ago

We just experienced a very similar issue with the Virtualbox provider. Vagrant never manages to connect but connecting via ssh just works. This issue prevents us from running our provision steps. Given that #12047 was already closed I added my comment here.

FYI we did update everything to the latest version and made sure the OSX permissions were all set. We rebooted as well to make sure the kernel modules were loaded correctly.

We have now resorted to downgrading back to Catalina so I cannot currently reproduce or run a debug session.

I sincerely hope by the time we all upgrade to Big Sur the transition will be smoother.

ryanmoon commented 3 years ago

I'm not sure the suggestion for using ssh_info_public is the solution, as we build all of our vagrant boxes with that flag as part of their Vagrantfile.

For instance, a macOS Big Sur box built with:

Vagrant.configure("2") do |config|
  config.vm.communicator = "ssh"
  config.vm.network :forwarded_port, guest: 22, host: 2222, id: "ssh", auto_correct:true
  ["vmware_fusion", "vmware_workstation", "vmware_desktop"].each do |provider|
    config.vm.provider provider do |vmware|
      vmware.gui = true
      vmware.ssh_info_public = true
      vmware.whitelist_verified = true
    end
  end
end

Still has the issue, granted this is a VMware Fusion issue moreso than a vagrant issue.

Vagrant: 2.2.13 vagrant-vmware-desktop: 2.1.5 VMware Fusion: 12.0,0 Test Kitchen: 2.7.2 macOS 11.0.1

4levels commented 3 years ago

@chrisroberts are you suggesting I should open another issue since upgrading to BigSur also breaks the Virtualbox provider as we experienced first hand?

chrisroberts commented 3 years ago

@ryanmoon Are you experiencing issues with Vagrant being unable to complete a vagrant up? If so, are you using the latest vagrant-vmware-utility release (1.0.16)?

@4levels No, a new issue shouldn't be opened as #12047 seems to cover the issue you are describing (it has just been closed prematurely).

michalmiddleton commented 3 years ago

For those interested in VMware NAT issue - Fusion 12.1.0 came out and the issue is still present. I just received the following response from VMware support:

NAT related issues in Fusion 12.1.0 are being the ones focused on the next maintenance release. There is background work going on for Jumbo Packets, NAT issues and most of the networking related issues. However we don't have an ETA as of now since we have to rely on Apple's response on the same.

craigontour commented 3 years ago

What about winrm? I am trying to use kitchen with vagrant to converge a W2012 VM (previously used VBox no problems). With VMWare Fusion 12 on Big Sur I just get:

==> default: Starting the VMware VM...
       ==> default: Waiting for the VM to receive an address...
       ==> default: Forwarding ports...
           default: -- 5985 => 55985
           default: -- 5986 => 55986
           default: -- 22 => 2222
       ==> default: Waiting for machine to boot. This may take a few minutes...
           default: WinRM address: 127.0.0.1:55985
           default: WinRM username: Administrator
           default: WinRM execution_time_limit: PT2H
           default: WinRM transport: negotiate
       Timed out while waiting for the machine to boot. This means that....

Is there a ssh_info_public equivalent for winrm ?

With 'winrm_info_public' I get error when i validate the ~/.vagrant.d/Vagrantfile:

VMware Desktop Provider:
* The following settings shouldn't exist: winrm_info_public
aleksf commented 3 years ago

ACK here. Hit the same issue on Catalina

Proposed workaround fixes the issue.

peerst commented 2 years ago

Found this because suddenly its hitting me too:

macOS Big Sur 11.6

VmwareFusion Professional Version 12.2.3

Tried the workaround but doesn't help.

bash-3.2$ vagrant up                                                             
Bringing machine 'default' up with 'vmware_fusion' provider...                   
==> default: Verifying vmnet devices are healthy...                              
==> default: Preparing network adapters...                                       
==> default: Starting the VMware VM...                                           
==> default: Waiting for the VM to receive an address...                         
==> default: Forwarding ports...                                                 
    default: -- 22 => 2222                                                       
==> default: Waiting for machine to boot. This may take a few minutes...         
    default: SSH address: 192.168.151.2:22                                       
    default: SSH username: vagrant                                               
    default: SSH auth method: private key                                        
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...                            
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.

If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.

If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.

If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.

Vagrantfile:

Vagrant.configure(2) do |config|
  config.ssh.forward_agent = true

  config.vm.box = "freebsd-i386-10.1-20G"

  config.vm.synced_folder ".", "/vagrant", 
  type: "nfs", :bsd__nfs_options => ["maproot=0:0"]

  config.vm.synced_folder  ENV['HOME'] + "/foobar", "/vagrant2", 
  type: "nfs", :bsd__nfs_options => ["maproot=0:0"]

  config.vm.synced_folder  ENV['HOME'] + "/packages-foobar", "/packages", 
  type: "nfs", :bsd__nfs_options => ["maproot=0:0"]

  config.vm.synced_folder  "../../archive", "/archive", 
  type: "nfs", :bsd__nfs_options => ["maproot=0:0"]

  config.vm.network "private_network", auto_config: false

  # config.vm.provision "shell", inline: "ifconfig em1 inet 192.168.188.14/24", run: "always"

  config.ssh.forward_x11 = true

  config.vm.provider "vmware_fusion" do |v|
    v.ssh_info_public = true
    v.gui = true
    v.vmx["memsize"] = "8192"
    v.vmx["numvcpus"] = "4"
  end

  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "playbook.yml"
    ansible.extra_vars = { 
      ansible_python_interpreter: '/usr/local/bin/python2.7' 
    }
  end

end

Yes I know this FreeBSD version is very old, its a legacy firmware for non internet connected devices. This VM is the only way I can maintain this system and now I'm locked out

peerst commented 2 years ago

Is this only on Big Sur and fixed on Monterey maybe?

peerst commented 2 years ago

Weirdly if I remove the ssh_info_public again I don't get "Connection refused" but it just hangs after this:

bash-3.2$ ssh vagrant@192.168.151.2
ssh: connect to host 192.168.151.2 port 22: Connection refused
bash-3.2$ ssh -i /Users/peer/projects/hydraprog-3/vagrant/box-i386/.vagrant/machines/default/vmware_fusion/private_key vagrant@192.168.151.2 
ssh: connect to host 192.168.151.2 port 22: Connection refused
bash-3.2$ vagrant ssh       
^Cbash-3.2$ vagrant halt
==> default: Attempting graceful shutdown of VM...
    default: Guest communication could not be established! This is usually because
    default: SSH is not running, the authentication information was changed,
    default: or some other networking issue. Vagrant will force halt, if
    default: capable.
==> default: Stopping the VMware VM...

bash-3.2$ 
bash-3.2$ vagrant up        
Bringing machine 'default' up with 'vmware_fusion' provider...
==> default: Verifying vmnet devices are healthy...
==> default: Preparing network adapters...
==> default: Starting the VMware VM...
==> default: Waiting for the VM to receive an address...
==> default: Forwarding ports...
    default: -- 22 => 2222
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
peerst commented 2 years ago

One more addendum: In both cases I can ssh into the vm with the internal guest IP address that ifconfig shows when I log into the console. Doesn't work as a workaround because I need the NFS mounts to happen and I don't know how to recreate them manually.

$ vagrant ssh-config
Host default
  HostName 192.168.151.2
  User vagrant
  Port 22
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /redacted
  IdentitiesOnly yes
  LogLevel FATAL
  ForwardAgent yes
  ForwardX11 yes

ifconfig on the inside shows: 192.168.151.128

$ ssh -I /redacted vagrant@192.168.151.128 works while $ ssh -I /redacted vagrant@192.168.151.2 does not.

The same goes for the case without ssh_info_public except that the file contains 120.0.0.1 and pork 2222 as expected. ssh only works with the guys internal IP

peerst commented 2 years ago

Worked around by statically setting the IP in the guest to 192.168.151.2 instead of DHCP otherwise while one can work around to be able to access the VM with ssh again it still breaks NFS where it stubbornly insists on this IP address.