sagetherage
commented
3 years ago Open ggiesen opened 3 years ago
sagetherage
commented
3 years ago 
ggiesen
commented
3 years ago @sagetherage Looks like I lied, this is still an issue. Tried firing up a VM today and had the same problem.
sagetherage
commented
3 years ago @dhiltonp we would love your take on this as well as we look at the VMware modules in Salt
sagetherage
commented
3 years ago @ggiesen a few of questions to help troubleshoot, here:
salt-cloud -a show_instance dc1newhost01 ?ggiesen
commented
3 years ago @ggiesen a few of questions to help troubleshoot, here:
* what does the /etc/salt/master.d/reactor.conf file look like? can you sanitize and share? * was the salt master restarted after this file was modified? * can you create a vm outside of a reactor with only a salt-cloud command such as `salt-cloud -a show_instance dc1newhost01` ?
/etc/salt/master.d/reactor.conf: reactor:
- 'salt/netapi/hook/netbox/virtualization/virtual-machines':
- 'salt://reactor/vmware_create.sls'
- 'salt/cloud/*/creating':
- 'salt://reactor/cloud_creating.sls'
- 'salt/cloud/*/requesting':
- salt://reactor/cloud_requesting.sls
- 'salt/cloud/*/querying':
- 'salt://reactor/cloud_querying.sls'
- 'salt/cloud/*/waiting_for_ssh':
- 'salt://reactor/cloud_waiting_for_ssh.sls'
- 'salt/cloud/*/deploying':
- 'salt://reactor/cloud_deploying.sls'
- 'salt/cloud/*/created':
- 'salt://reactor/cloud_created.sls'
- 'salt/cloud/*/destroying':
- 'salt://reactor/cloud_destroying.sls'
- 'salt/cloud/*/destroyed':
- 'salt://reactor/cloud_destroyed.sls'Yes, absolutely the master has been started many times since this file was modified. I'll try creating a VM outside of a reactor and see if I get the result.
ggiesen
commented
3 years ago Just to be clear, the VM is created correctly by the salt/netapi/hook/netbox/virtualization/virtual-machines reactor (triggered by a webhook from Netbox), it's just when it executes the code in salt://reactor/cloud_created.sls to gather things like the assigned MAC address (so that I can populate them back into Netbox), it circles around my loop for 20-30 mins, complaining it can't locate the VM.
ggiesen
commented
3 years ago If I run 'salt-cloud -a show_instance' on a VM that has been running for a while, it does indeed work. I'll try it on one that is newly-created and report back
ggiesen
commented
3 years ago I can also confirm on a newly-created VM, 'salt-cloud -a show_instance
sagetherage
commented
3 years ago It does sound like there is a problem and we see that it can take longer than expected at times, and we are not sure if is something we can fix, but we will try. We will review this again in triage tmr and attempt to get a development instance of vcenter to test it - we are still learning how we gain access to an instance, so I apologize for the delay, and I have not forgotten! :)
wegenerbenjamin
commented
3 years ago We have the same problem. The following change (line 476 in salt/utils/vmware.py) works as a workaround for us:
- except vim.fault.NotAuthenticated:
+ except (vim.fault.NotAuthenticated, ssl.SSLError, BrokenPipeError):
vkotarov
commented
1 year ago Added ConnectionResetError to the list in order to address very rare cases where cloud failed. In my case I am executing cloud client from a runner and I get BrokenPipeError on almost every attempt.
a-wildman
commented
10 months ago Confirmed this is still an issue in 3006.3
Description When creating a new VM using the vmware driver in salt-cloud, I'm using a reactor python script to fetch VM details using the salt/cloud/*/created reactor. I've tried querying using both the python client API:
(vmware_data = cclient.action(fun="show_instance", instance=data["name"])as well as the internal salt function:
vmware_data = cclient.action(fun="show_instance", instance=data["name"])I have four VMware providers defined (4 different DCs), and when running the reactor script, I get:
Regardless of which DC I provision the new VM in, I will get messages about the host not being able to be located in the other 3 DCs.
I can however run salt-cloud to query the instance (while debug is still looping trying to find the VM) and it works perfectly:
Here's a code snippet from my reactor:
With the loop I've implemented, it will eventually (20-30 mins later) finally complete running the reactor and provide the requested data
Setup /etc/salt/cloud.providers.d/vmware.conf:
DC1: driver: vmware user: sdb://secret/vsphere:username password: sdb://secret/vsphere:password url: 'dc1.example.com' protocol: 'https' port: 443 DC2: driver: vmware user: sdb://secret/vsphere:username password: sdb://secret/vsphere:password url: 'dc2.example.com' protocol: 'https' port: 443 DC3: driver: vmware user: sdb://secret/vsphere:username password: sdb://secret/vsphere:password url: 'dc3.example.com protocol: 'https' port: 443 DC4: driver: vmware user: sdb://secret/vsphere:username password: sdb://secret/vsphere:password url: 'dc4.example.com' protocol: 'https' port: 443
Steps to Reproduce the behavior Create a new vm using the vmware driver, and then set up a reactor with the code above
Expected behavior VM locates vm and cloud provider returns requested data
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ``` Salt Version: Salt: 3001.1 Dependency Versions: cffi: 1.14.0 cherrypy: unknown dateutil: 2.6.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.11.2 libgit2: 1.0.1 M2Crypto: 0.35.2 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.6.2 mysql-python: Not Installed pycparser: 2.20 pycrypto: Not Installed pycryptodome: Not Installed pygit2: 1.3.0 Python: 3.6.8 (default, Apr 16 2020, 01:36:27) python-gnupg: 0.4.6 PyYAML: 5.3.1 PyZMQ: 19.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.3 System Versions: dist: centos 8 Core locale: UTF-8 machine: x86_64 release: 4.18.0-193.19.1.el8_2.x86_64 system: Linux version: CentOS Linux 8 Core ```