search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.12k stars 5.47k forks source link

Azure: salt-cloud -d <vm-name> doesn't work. #15958

Closed egilh closed 6 years ago

egilh commented 10 years ago

I have an issue where VM's are created, but no hostname is returned to the master. This means no salt installation occurs and the minion has to be added manually. When trying to delete the VM I get an error saying no VM's where found.

            Salt: 2014.7.0rc2
          Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
          Jinja2: 2.7.2
        M2Crypto: 0.21.1
  msgpack-python: 0.3.0
    msgpack-pure: Not Installed
        pycrypto: 2.6.1
         libnacl: Not Installed
          PyYAML: 3.10
           ioflo: Not Installed
           PyZMQ: 14.0.1
            RAET: Not Installed
             ZMQ: 4.0.4
            Mako: 0.9.1
 Apache Libcloud: 0.14.1
root@salt-master:# salt-cloud -d yrdocker2
[DEBUG   ] Configuration file path: /etc/salt/cloud
[INFO    ] salt-cloud starting
[DEBUG   ] There is no IBM SCE cloud provider configuration available. Not loading module.
[DEBUG   ] 'parallels.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'parallels.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'proxmox.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.destroy' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_images' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'rackspace.reboot' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] The 'azure' cloud driver is unable to be optimized.
[DEBUG   ] There is no IBM SCE cloud provider configuration available. Not loading module.
[DEBUG   ] 'parallels.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'parallels.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'proxmox.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.destroy' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_images' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'rackspace.reboot' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] Failed to execute 'azure.list_nodes()' while querying for running nodes: Not found (Not Found)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2195, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 240, in list_nodes
    nodes = list_nodes_full(conn, call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 259, in list_nodes_full
    services = list_hosted_services(conn=conn, call=call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 361, in list_hosted_services
    role_name=role.role_name,
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementservice.py", line 997, in get_role
    PersistentVMRole)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 108, in _perform_get
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 97, in _perform_request
    return _management_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/__init__.py", line 1313, in _management_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 894, in _general_error_handler
    _ERROR_NOT_FOUND.format(str(http_error)))
WindowsAzureMissingResourceError: Not found (Not Found)
No machines were found to be destroyed
rallytime commented 10 years ago

Thanks for the report @egilh. I have not seen this behavior before on 2014.7 branch, and have tests running nightly to check for the creation and deletion of VMs on Azure. What kinds of errors do you get when you create the VM? Also, can you also test this same sequence (creating the VM with those issues, and then have the -d problems) on the most recent version of 2014.7 (instead of rc2)?

egilh commented 10 years ago

@rallytime thanks for the reply. I've just tried running 2014.7 on Ubuntu 12.04. Running salt-cloud -p azure-ubuntu I still get the same errors as before. Provisioning doesn't get a hostname returned, which means deletion doesnt work either.

yrdevops@salt-master2:~$ salt-cloud --versions-report
            Salt: 2014.7.0
          Python: 2.7.3 (default, Feb 27 2014, 19:58:35)
          Jinja2: 2.6
        M2Crypto: 0.21.1
  msgpack-python: 0.1.10
    msgpack-pure: Not Installed
        pycrypto: 2.4.1
         libnacl: Not Installed
          PyYAML: 3.10
           ioflo: Not Installed
           PyZMQ: 13.0.0
            RAET: Not Installed
             ZMQ: 3.2.2
 Apache Libcloud: 0.15.1

Trying to provision VM: (ctrl-c after a while, cannot be bothered to wait)

~$ sudo salt-cloud -p ubuntu_1204 1204salt
[DEBUG   ] Configuration file path: /etc/salt/cloud
[INFO    ] salt-cloud starting
[DEBUG   ] 'parallels.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'parallels.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'proxmox.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.destroy' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_images' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'rackspace.reboot' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] The 'azure' cloud driver is unable to be optimized.
[DEBUG   ] 'parallels.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'parallels.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'proxmox.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.destroy' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_images' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'rackspace.reboot' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] Failed to execute 'azure.list_nodes()' while querying for running nodes: Not found (Not Found)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2166, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 240, in list_nodes
    nodes = list_nodes_full(conn, call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 259, in list_nodes_full
    services = list_hosted_services(conn=conn, call=call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 360, in list_hosted_services
    role_name=role.role_name,
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementservice.py", line 997, in get_role
    PersistentVMRole)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 108, in _perform_get
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 97, in _perform_request
    return _management_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/__init__.py", line 1313, in _management_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 894, in _general_error_handler
    _ERROR_NOT_FOUND.format(str(http_error)))
WindowsAzureMissingResourceError: Not found (Not Found)
[DEBUG   ] Generating minion keys for '1204salt'
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'profile': 'ubuntu_1204', 'event': 'starting create', '_stamp': '2014-09-22T09:40:51.034579', 'name': '1204salt', 'provider': 'yr-azure-config:azure'}
[INFO    ] Creating Cloud VM 1204salt
[DEBUG   ] vm_kwargs: {'system_config': <azure.servicemanagement.LinuxConfigurationSet object at 0x3896d90>, 'deployment_slot': 'production', 'role_size': 'Small', 'deployment_name': '1204salt', 'service_name': '1204salt', 'role_name': '1204salt', 'network_config': <azure.servicemanagement.ConfigurationSet object at 0x2fba690>, 'os_virtual_hard_disk': <azure.servicemanagement.OSVirtualHardDisk object at 0x3896a10>, 'label': '1204salt'}
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2014-09-22T09:40:51.093374', 'service_kwargs': {'service_name': '1204salt', 'label': '1204salt', 'location': 'West Europe', 'description': '1204salt'}, 'event': 'requesting instance', 'vm_kwargs': {'deployment_slot': 'production', 'role_size': 'Small', 'deployment_name': '1204salt', 'service_name': '1204salt', 'label': '1204salt', 'role_name': '1204salt'}}
[DEBUG   ] vm_kwargs: {'system_config': <azure.servicemanagement.LinuxConfigurationSet object at 0x3896d90>, 'deployment_slot': 'production', 'role_size': 'Small', 'deployment_name': '1204salt', 'service_name': '1204salt', 'role_name': '1204salt', 'network_config': <azure.servicemanagement.ConfigurationSet object at 0x2fba690>, 'os_virtual_hard_disk': <azure.servicemanagement.OSVirtualHardDisk object at 0x3896a10>, 'label': '1204salt'}
[DEBUG   ] Attempting function <function wait_for_hostname at 0x2e56578>
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 1)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 2)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 3)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 4)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 5)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 6)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 7)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 8)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 9)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 10)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 11)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 12)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 13)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 14)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 15)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 16)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 17)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 18)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 19)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 20)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 21)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 22)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 23)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 24)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 25)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 26)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 27)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 28)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 29)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 30)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 31)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 32)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 33)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 34)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 35)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 36)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 37)
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e56578> on  (try 38)
^C
Exiting gracefully on Ctrl-c

Trying to delete the VM:

~$ sudo salt-cloud -d 1204salt
[DEBUG   ] Configuration file path: /etc/salt/cloud
[INFO    ] salt-cloud starting
[DEBUG   ] 'parallels.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'parallels.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'proxmox.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.destroy' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_images' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'rackspace.reboot' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] The 'azure' cloud driver is unable to be optimized.
[DEBUG   ] 'parallels.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'parallels.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'proxmox.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.destroy' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_images' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'rackspace.reboot' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] Failed to execute 'azure.list_nodes()' while querying for running nodes: Not found (Not Found)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2166, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 240, in list_nodes
    nodes = list_nodes_full(conn, call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 259, in list_nodes_full
    services = list_hosted_services(conn=conn, call=call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 360, in list_hosted_services
    role_name=role.role_name,
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementservice.py", line 997, in get_role
    PersistentVMRole)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 108, in _perform_get
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 97, in _perform_request
    return _management_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/__init__.py", line 1313, in _management_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 894, in _general_error_handler
    _ERROR_NOT_FOUND.format(str(http_error)))
WindowsAzureMissingResourceError: Not found (Not Found)
No machines were found to be destroyed

The VM is there, its has my certificate and I can log in:

data:    1204salt      ReadyRole  West Europe  1204salt.cloudapp.net      100.74.246.43
rallytime commented 10 years ago

Thanks for the update @egilh. Have you been able to create and destroy VMs with Azure on a different release, such as 2014.1.10? Can you get information back from Azure when you do a salt-cloud --list-sizes or other such commands?

egilh commented 10 years ago

@rallytime I have had no luck 'successfully' provisioning, because I get the same error using 2014.1.10, 2014.7.0rc2 and 2014.7. That meaning that the VM's are up and running, but without being bootstrapped. I can list images and list sizes:

s# salt-cloud --list-images azure | grep 'Ubuntu-14'
[DEBUG   ] Configuration file path: /etc/salt/cloud
[INFO    ] salt-cloud starting
[DEBUG   ] There is no IBM SCE cloud provider configuration available. Not loading module.
[DEBUG   ] 'parallels.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'parallels.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'proxmox.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.destroy' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_images' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'rackspace.reboot' has been marked as not supported. Removing from the list of supported cloud functions
        0b11de9248dd4d87b18621318e037d37__RightImage-Ubuntu-14.04-x64-v14.1:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140414-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140414.2-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140416.1-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140528-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140606.1-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140618.1-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140724-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_1-LTS-amd64-server-20140909-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_10-amd64-server-20140625-alpha1-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_10-amd64-server-20140729-alpha2-en-us-30GB:
        b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_10-amd64-server-20140826-beta1-en-us-30GB:

The thing that gets me though, is that I get this stacktrace every time:

[DEBUG   ] Failed to execute 'azure.list_nodes_full()' while querying for running nodes: Not found (Not Found)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2195, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 259, in list_nodes_full
    services = list_hosted_services(conn=conn, call=call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 361, in list_hosted_services
    role_name=role.role_name,
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementservice.py", line 997, in get_role
    PersistentVMRole)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 108, in _perform_get
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 97, in _perform_request
    return _management_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/__init__.py", line 1313, in _management_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 894, in _general_error_handler
    _ERROR_NOT_FOUND.format(str(http_error)))
WindowsAzureMissingResourceError: Not found (Not Found)

I was planning to use SaltStack for provisioning, orchestrating and deploying to Azure, but I've spent a week trying to troubleshoot, and I'm really stuck here. I have followed the docs, everything seems in order.

egilh commented 10 years ago

I am really starting to look at list_nodes here. It seems that this is whats failing for me, all the time. Tried running develop brach, no luck there either:

# salt-cloud --versions-report
            Salt: 2014.7.0rc2-1742-ge608e8c
          Python: 2.7.3 (default, Feb 27 2014, 19:58:35)
          Jinja2: 2.6
        M2Crypto: 0.21.1
  msgpack-python: 0.1.10
    msgpack-pure: Not Installed
        pycrypto: 2.4.1
         libnacl: Not Installed
          PyYAML: 3.10
           ioflo: Not Installed
           PyZMQ: 13.0.0
            RAET: Not Installed
             ZMQ: 3.2.2
            Mako: Not Installed
 Apache Libcloud: 0.15.1

Trying to run salt-cloud -f list_nodes azure:

# salt-cloud -f -list_nodes azure
[DEBUG   ] Configuration file path: /etc/salt/cloud
[INFO    ] salt-cloud starting
[DEBUG   ] Failed to import cloud pyrax, this is most likely NOT a problem:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 896, in load_modules
    ), fn_, path, desc
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/pyrax.py", line 18, in <module>
    import salt.utils.openstack.pyrax as suop
ImportError: No module named pyrax
Error: There was an error running the function: The 'yr-azure-config' cloud provider alias, for the 'azure' driver, does not define the function '-list_nodes'

Trying to provision:

# salt-cloud -p azure-ubuntu salt12345
[DEBUG   ] Configuration file path: /etc/salt/cloud
[INFO    ] salt-cloud starting
[DEBUG   ] Failed to import cloud pyrax, this is most likely NOT a problem:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 896, in load_modules
    ), fn_, path, desc
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/pyrax.py", line 18, in <module>
    import salt.utils.openstack.pyrax as suop
ImportError: No module named pyrax
[DEBUG   ] The 'azure' cloud driver is unable to be optimized.
[DEBUG   ] Failed to import cloud pyrax, this is most likely NOT a problem:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 896, in load_modules
    ), fn_, path, desc
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/pyrax.py", line 18, in <module>
    import salt.utils.openstack.pyrax as suop
ImportError: No module named pyrax
[DEBUG   ] Failed to execute 'azure.list_nodes()' while querying for running nodes: Not found (Not Found)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2226, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 240, in list_nodes
    nodes = list_nodes_full(conn, call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 259, in list_nodes_full
    services = list_hosted_services(conn=conn, call=call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 361, in list_hosted_services
    role_name=role.role_name,
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementservice.py", line 997, in get_role
    PersistentVMRole)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 108, in _perform_get
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 97, in _perform_request
    return _management_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/__init__.py", line 1336, in _management_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 910, in _general_error_handler
    _ERROR_NOT_FOUND.format(str(http_error)))
WindowsAzureMissingResourceError: Not found (Not Found)
[DEBUG   ] Generating minion keys for 'salt12345'
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'profile': 'azure-ubuntu', 'event': 'starting create', '_stamp': '2014-09-23T08:32:20.013791', 'name': 'salt12345', 'provider': 'yr-azure-config:azure'}
[INFO    ] Creating Cloud VM salt12345
[DEBUG   ] vm_kwargs: {'system_config': <azure.servicemanagement.LinuxConfigurationSet object at 0x259ecd0>, 'deployment_slot': 'production', 'role_size': 'Small', 'deployment_name': 'salt12345', 'service_name': 'salt12345', 'role_name': 'salt12345', 'network_config': <azure.servicemanagement.ConfigurationSet object at 0x2e8d3d0>, 'os_virtual_hard_disk': <azure.servicemanagement.OSVirtualHardDisk object at 0x2e7d450>, 'label': 'salt12345'}
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2014-09-23T08:32:25.080475', 'service_kwargs': {'service_name': 'salt12345', 'label': 'salt12345', 'location': 'West Europe', 'description': 'salt12345'}, 'event': 'requesting instance', 'vm_kwargs': {'deployment_slot': 'production', 'role_size': 'Small', 'deployment_name': 'salt12345', 'service_name': 'salt12345', 'label': 'salt12345', 'role_name': 'salt12345'}}
[DEBUG   ] vm_kwargs: {'system_config': <azure.servicemanagement.LinuxConfigurationSet object at 0x259ecd0>, 'deployment_slot': 'production', 'role_size': 'Small', 'deployment_name': 'salt12345', 'service_name': 'salt12345', 'role_name': 'salt12345', 'network_config': <azure.servicemanagement.ConfigurationSet object at 0x2e8d3d0>, 'os_virtual_hard_disk': <azure.servicemanagement.OSVirtualHardDisk object at 0x2e7d450>, 'label': 'salt12345'}
[DEBUG   ] Attempting function <function wait_for_hostname at 0x2e685f0>
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x2e685f0> on  (try 1)
^C  
Exiting gracefully on Ctrl-c

Again, the VM is deployed:

+ Getting virtual machines                                                     
data:    Name          Status     Location     DNS Name                   IP Address    
data:    ------------  ---------  -----------  -------------------------  --------------
data:    salt12345     ReadyRole  West Europe  salt12345.cloudapp.net     100.74.244.168

Trying to delete the VM:

# salt-cloud -d salt12345
[DEBUG   ] Configuration file path: /etc/salt/cloud
[INFO    ] salt-cloud starting
[DEBUG   ] Failed to import cloud pyrax, this is most likely NOT a problem:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 896, in load_modules
    ), fn_, path, desc
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/pyrax.py", line 18, in <module>
    import salt.utils.openstack.pyrax as suop
ImportError: No module named pyrax
[DEBUG   ] The 'azure' cloud driver is unable to be optimized.
[DEBUG   ] Failed to import cloud pyrax, this is most likely NOT a problem:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 896, in load_modules
    ), fn_, path, desc
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/pyrax.py", line 18, in <module>
    import salt.utils.openstack.pyrax as suop
ImportError: No module named pyrax
[DEBUG   ] Failed to execute 'azure.list_nodes()' while querying for running nodes: Not found (Not Found)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2226, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 240, in list_nodes
    nodes = list_nodes_full(conn, call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 259, in list_nodes_full
    services = list_hosted_services(conn=conn, call=call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 361, in list_hosted_services
    role_name=role.role_name,
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementservice.py", line 997, in get_role
    PersistentVMRole)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 108, in _perform_get
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 97, in _perform_request
    return _management_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/__init__.py", line 1336, in _management_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 910, in _general_error_handler
    _ERROR_NOT_FOUND.format(str(http_error)))
WindowsAzureMissingResourceError: Not found (Not Found)
No machines were found to be destroyed

Guess this really has not so much to with '-d' specificlly,but azure.list_nodes not working.

rallytime commented 10 years ago

Thanks for the additional info. We'll definitely take a look at this, however I can't seem to reproduce this problem.

@techhat or @s0undt3ch ping.

egilh commented 10 years ago

Let me know if any additional info is needed. I've tried running on both Ubuntu 12.04 and 14.04, both local master and a master running in Azure. I get the same results across the board.

I am also going to try to provision some Windows hosts, but I am seeing some weird results there also. I'll report back here if I think its related.

egilh commented 10 years ago

It can seem that this is caused by one or two different things. 1) It seems I get a 404 from Azure. We have one service deployed, that uses dots '.' in the deploy-name and. This seems to get Salt confused by the response from Azure as it lists the services in order, and it fails from there. Hence the 'list_nodes() failure. 2) That there's a bug in the SDK or the way Salt sends the commands to Azure: My requests that fails:

{subscription-id}/services/hostedservices/{service-name}/deployments/{deployment-name}/roles/{role-name}/Operations

Proper one, according to Microsoft:

{subscriptionID}/services/hostedservices/{serviceName}/deployments/{deploymentName}/roleInstances/{roleInstanceName}/Operations

I can't understand that I'm the only one seeing this issue. @rallytime Can you try to set up a VM with a '.' in deployment name, and try to provision from SaltStack again? I tried to debug some more and print out whats happening:

App_debug: management.core.windows.net /<subscription-id>/services/hostedservices
App_debug: management.core.windows.net /<subscription-id>/services/hostedservices/salt-master?embed-detail=true
App_debug: management.core.windows.net /<subscription-id>/services/hostedservices/salt-master/deployments/salt-master/roles/salt-master
App_debug: management.core.windows.net /<subscription-id>/services/hostedservices/salt-master2?embed-detail=true
App_debug: management.core.windows.net /<subscription-id>/services/hostedservices/salt-master2/deployments/salt-master2/roles/salt-master2
App_debug: management.core.windows.net /<subscription-id>/services/hostedservices/stadnafn?embed-detail=true
App_debug: management.core.windows.net /<subscription-id>/services/hostedservices/stadnafn/deployments/5e6663e032b541feb27e9f41c44ab65a/roles/Nrk.Stadnafn.WebApi
[DEBUG   ] Failed to execute 'azure.list_nodes()' while querying for running nodes: Not found (Not Found)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2226, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 240, in list_nodes
    nodes = list_nodes_full(conn, call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 259, in list_nodes_full
    services = list_hosted_services(conn=conn, call=call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 361, in list_hosted_services
    role_name=role.role_name,
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementservice.py", line 997, in get_role
    PersistentVMRole)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 110, in _perform_get
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 97, in _perform_request
    return _management_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/__init__.py", line 1336, in _management_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 910, in _general_error_handler
    _ERROR_NOT_FOUND.format(str(http_error)))
WindowsAzureMissingResourceError: Not found (Not Found)

If I'm correct, this is a rather nasty bug that should be easily fixed.

rallytime commented 10 years ago

Yep! I will try and deploy an instance with a . in the name shortly.

rallytime commented 10 years ago

Thanks for your detective work here @egilh! You are correct in that putting a . in the instance name causes this error. Some cloud providers support putting dots in VM names and some do not. Azure does not.

From the Azure web interface: "The hosted service name is invalid. The name can contain only letters, numbers, and hyphens. The name must start with a letter and must end with a letter or a number."

So, the fix here should be to handle this error more elegantly. We shouldn't be stacktracing, and we should be letting you know that you can't use a . in the instance name.

egilh commented 10 years ago

@rallytime thanks for double-checking for me. The thing here though, is that there is no VM with a '.' in the VM name, I am not trying to provision a VM with a '.'the name either, in fact, this is a Cloud Service and not even a VM.

Salt should be able to ignore the '.' when getting running nodes returned. I guess my workaround for right now, will be to try and deploy that Cloud Service with another name, but I really think this is an Salt or Azure SDK issue, and has nothing to do with Azure on its own.

basepi commented 10 years ago

@egilh If you're not trying to provision a VM with the . in the name, then how is the . getting there?

basepi commented 10 years ago

Also, I don't think you can have dots in the cloud service name either.

egilh commented 10 years ago

@basepi I thought I had specified this above. The '.' comes from Azure, because Salt tries to run list_nodes during provisioning. This fails, because we have a Cloud Service with a deployment-label cointaining a '.'. The name of the cloud-service itself does not. (I might have called it deployment-name above, my mistake)

Azure makes a difference of VM-Name, Cloud-Service Name, Deployment Name, Deployment Label and Deployment ID. In fact, salt-cloud should also be able to know the difference. Right now, it seems to belivie they are all the same.

egilh commented 10 years ago

And also, a cloud service in Azure doesn't need a VM to be running.

rallytime commented 10 years ago

How were you able to create a Cloud Service with '.' in the name? If I try to create a cloud service on Azure with a '.' in the name, I get the following error:

"This field can contain only letters, numbers, and hyphens. The first and last character in the field must be a letter or number. Trademarks, reserved words, and offensive words are not allowed."

rallytime commented 10 years ago

Also, are you using a map file to create your instances?

egilh commented 10 years ago

@rallytime I don't have a Cloud Service with a . in the name, the deployment label of the cloud service does. This gets returned in the request.

I see that salt-cloud sees both service, label and name as the same, but in Azure, they can be different. Its a completley normal way to deploy Cloud Services to Azure, containing . in the Deployment LABEL of the service. A service can be upgraded, deployed to, and so on, regardless of VM's.

Sorry if I'm not being clear here, but this is the way Azure works.

egilh commented 10 years ago

And no, I have had no luck using map files, because it stacktraces on the same 'list_nodes', which means that when deploying a map file, only the first (or second) VM in the map file gets deployed. Again, it fails because Azure returns all running Cloud Servics to Salt, and one of my Cloud Services have a Deployment with a Deployment Label containing a .

egilh commented 10 years ago

Here's a complete debug log from trying to provision a VM to Azure, named saltdebug2. . I've added a debug print from servicemangement.py so I can see the response from Azure that salt stacktraces on. You can see that it retries over and over again, but stops on the same response, the Cloud Service with a Deployment Label containing a . I cancelled it after the VM was deployed. Salt-Cloud was still hanging on the same request. (This is using develop branch though, but same thing happens on stable and rc)

# salt-cloud -p azure-ubuntu saltdebug3
[DEBUG   ] Configuration file path: /etc/salt/cloud
[INFO    ] salt-cloud starting
[DEBUG   ] Failed to import cloud pyrax, this is most likely NOT a problem:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 897, in load_modules
    ), fn_, path, desc
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/pyrax.py", line 18, in <module>
    import salt.utils.openstack.pyrax as suop
ImportError: No module named pyrax
[DEBUG   ] Failed to import cloud digital_ocean_v2, this is most likely NOT a problem:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 897, in load_modules
    ), fn_, path, desc
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/digital_ocean_v2.py", line 34, in <module>
    from salt.exceptions import (
ImportError: cannot import name SaltCloudConfigError
[DEBUG   ] 'parallels.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'parallels.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'proxmox.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.destroy' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_images' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'rackspace.reboot' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] The 'azure' cloud driver is unable to be optimized.
[DEBUG   ] Failed to import cloud pyrax, this is most likely NOT a problem:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 897, in load_modules
    ), fn_, path, desc
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/pyrax.py", line 18, in <module>
    import salt.utils.openstack.pyrax as suop
ImportError: No module named pyrax
[DEBUG   ] Failed to import cloud digital_ocean_v2, this is most likely NOT a problem:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 897, in load_modules
    ), fn_, path, desc
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/digital_ocean_v2.py", line 34, in <module>
    from salt.exceptions import (
ImportError: cannot import name SaltCloudConfigError
[DEBUG   ] 'parallels.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'parallels.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'proxmox.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.destroy' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_sizes' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_images' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'saltify.avail_locations' has been marked as not supported. Removing from the list of supported cloud functions
[DEBUG   ] 'rackspace.reboot' has been marked as not supported. Removing from the list of supported cloud functions
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master/deployments/salt-master/roles/salt-master
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2/deployments/salt-master2/roles/salt-master2
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn/deployments/5e6663e032b541feb27e9f41c44ab65a/roles/Nrk.Stadnafn.WebApi
[DEBUG   ] Failed to execute 'azure.list_nodes()' while querying for running nodes: Not found (Not Found)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 2166, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 240, in list_nodes
    nodes = list_nodes_full(conn, call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 259, in list_nodes_full
    services = list_hosted_services(conn=conn, call=call)
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 360, in list_hosted_services
    role_name=role.role_name,
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementservice.py", line 997, in get_role
    PersistentVMRole)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 110, in _perform_get
    response = self._perform_request(request)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/servicemanagementclient.py", line 97, in _perform_request
    return _management_error_handler(ex)
  File "/usr/local/lib/python2.7/dist-packages/azure/servicemanagement/__init__.py", line 1336, in _management_error_handler
    return _general_error_handler(http_error)
  File "/usr/local/lib/python2.7/dist-packages/azure/__init__.py", line 910, in _general_error_handler
    _ERROR_NOT_FOUND.format(str(http_error)))
WindowsAzureMissingResourceError: Not found (Not Found)
[DEBUG   ] Generating minion keys for 'saltdebug3'
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'profile': 'azure-ubuntu', 'event': 'starting create', '_stamp': '2014-09-24T19:25:02.288002', 'name': 'saltdebug3', 'provider': 'yr-azure-config:azure'}
[INFO    ] Creating Cloud VM saltdebug3
[DEBUG   ] vm_kwargs: {'system_config': <azure.servicemanagement.LinuxConfigurationSet object at 0x2a09f50>, 'deployment_slot': 'production', 'role_size': 'Small', 'deployment_name': 'saltdebug3', 'service_name': 'saltdebug3', 'role_name': 'saltdebug3', 'network_config': <azure.servicemanagement.ConfigurationSet object at 0x2a09e90>, 'os_virtual_hard_disk': <azure.servicemanagement.OSVirtualHardDisk object at 0x2a09690>, 'label': 'saltdebug3'}
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'_stamp': '2014-09-24T19:25:07.362360', 'service_kwargs': {'service_name': 'saltdebug3', 'label': 'saltdebug3', 'location': 'West Europe', 'description': 'saltdebug3'}, 'event': 'requesting instance', 'vm_kwargs': {'deployment_slot': 'production', 'role_size': 'Small', 'deployment_name': 'saltdebug3', 'service_name': 'saltdebug3', 'label': 'saltdebug3', 'role_name': 'saltdebug3'}}
[DEBUG   ] vm_kwargs: {'system_config': <azure.servicemanagement.LinuxConfigurationSet object at 0x2a09f50>, 'deployment_slot': 'production', 'role_size': 'Small', 'deployment_name': 'saltdebug3', 'service_name': 'saltdebug3', 'role_name': 'saltdebug3', 'network_config': <azure.servicemanagement.ConfigurationSet object at 0x2a09e90>, 'os_virtual_hard_disk': <azure.servicemanagement.OSVirtualHardDisk object at 0x2a09690>, 'label': 'saltdebug3'}
[DEBUG   ] Attempting function <function wait_for_hostname at 0x29fac80>
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master/deployments/salt-master/roles/salt-master
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2/deployments/salt-master2/roles/salt-master2
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/saltdebug3?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn/deployments/5e6663e032b541feb27e9f41c44ab65a/roles/Nrk.Stadnafn.WebApi
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x29fac80> on  (try 1)
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master/deployments/salt-master/roles/salt-master
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2/deployments/salt-master2/roles/salt-master2
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/saltdebug3?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/saltdebug3/deployments/saltdebug3/roles/saltdebug3
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn/deployments/5e6663e032b541feb27e9f41c44ab65a/roles/Nrk.Stadnafn.WebApi
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x29fac80> on  (try 2)
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master/deployments/salt-master/roles/salt-master
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2/deployments/salt-master2/roles/salt-master2
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/saltdebug3?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/saltdebug3/deployments/saltdebug3/roles/saltdebug3
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn/deployments/5e6663e032b541feb27e9f41c44ab65a/roles/Nrk.Stadnafn.WebApi
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x29fac80> on  (try 3)
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master/deployments/salt-master/roles/salt-master
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2/deployments/salt-master2/roles/salt-master2
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/saltdebug3?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/saltdebug3/deployments/saltdebug3/roles/saltdebug3
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn/deployments/5e6663e032b541feb27e9f41c44ab65a/roles/Nrk.Stadnafn.WebApi
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x29fac80> on  (try 4)
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master/deployments/salt-master/roles/salt-master
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2/deployments/salt-master2/roles/salt-master2
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/saltdebug3?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/saltdebug3/deployments/saltdebug3/roles/saltdebug3
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/stadnafn/deployments/5e6663e032b541feb27e9f41c44ab65a/roles/Nrk.Stadnafn.WebApi
[DEBUG   ] Caught exception in wait_for_fun: local variable 'data' referenced before assignment
[DEBUG   ] Retrying function <function wait_for_hostname at 0x29fac80> on  (try 5)
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master?embed-detail=true
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master/deployments/salt-master/roles/salt-master
App_debug: management.core.windows.net /5f55cd86-65d7-46c9-b989-06798b2c480f/services/hostedservices/salt-master2?embed-detail=true
^C
Exiting gracefully on Ctrl-c
rallytime commented 10 years ago

Oh can you try this same thing on a new version of 2014.7? Deploying on Azure is indeed broken on develop. I filed an issue about that yesterday: #16074

rallytime commented 10 years ago

Actually you're still hitting the same problem. So you can scratch running this again.

rallytime commented 10 years ago

Thanks for your patience and all of the examples. I have only recently been working with Azure on Salt Cloud, so I am not familiar with where the "Deployment Label" is getting set so I can try to reproduce this issue.

egilh commented 10 years ago

@rallytime I'm not sure atm how to do that using Azure SDK or Azure xcli. I can see if I can figure that one out. This particular solution is deployed directly from VisualStudio, and therefor gets the same deployment-label as the solution-name in VS. I also know you can set it using PowerShell during deploy, but I'm not sure if it can be changed after deploy. Trying to figure it out right now .

rallytime commented 10 years ago

That's totally fine. I just wasn't sure where that value was coming from, i.e. if you were setting some value in your cloud provider/profile files or doing something on the Azure web interface. The information you've provided is helpful.

egilh commented 10 years ago

Not sure if these two are related, or if I've been wrong here and its not the . at all. It can be caused by role-instances in cloud-services. Check #16140

egilh commented 10 years ago

@rallytime I might have been a little quick on the trigger here, claiming its the .

I've made a real ugly workaround that excludes all the cloud services that gives me trouble, and it could seem that it could be a problem with cloud-services that have instances that are not VM's. I can't really double-check this myself, but one can use Azure-cli to deploy a service without a VM.

egilh commented 9 years ago

Any progress here? Salt is completly useless for me in Azure, as long as this doesn't work.

rallytime commented 9 years ago

@egilh Thanks for the bump and apologies for the delay. I don't think any one has had a chance to look at this issue yet. We've got a substantial bug count that we are working hard to tackle. We'll get to this one, too, as soon as we can.

andrejohansson commented 9 years ago

This is not only caused by dots in the name. I've come across some other issue (don't remember which now but it's here somewhere) where list_nodes fails if you have vms with no current ips assigned to them (ie they are shut of) or something. There seems to be a lot of responses causing list_nodes to fail, and when that happens...pretty much all crucial azure commands stop working.

Also, since one of these conditions need to be met, when devs try to reproduce I guess they do it on quite blank Azure subscriptions and things work fine.

My suggested solution is to work in the direction to make list_nodes be more tolerant of different responses and exit gracefully or with "as much info as possible" instead of just crashing since it blocks most commands.

And yes, I understand its kind of a vague suggestion...sorry for that :-)

egilh commented 9 years ago

Like @andrejohansson says, it seems this happens because of several reasons.

The reason it happens for me, every time, is because we are already running in a Azure enviroment with several Cloud Services, that don't contain Virtual Machines. This should be easy to reproduce as I see it, unfortunatly, I dont have another Azure env. to test it on myself.

rallytime commented 9 years ago

I know that @techhat did a lot of work with Azure lately, though I don't know if any of that work was addressing this issue specifically. Has anyone tried this on a newer version of salt? I suspect this is still an issue, but thought I would get some feedback regardless. Thanks!

techhat commented 9 years ago

This may be resolved in the develop branch. Please check http://docs.saltstack.com/en/develop/topics/cloud/azure.html#destroying-vms

rallytime commented 9 years ago

Was anyone able to give the develop version a try to see if this is fixed?

bradthurber commented 9 years ago

@rallytime I just started working with Azure and salt-cloud v2015.5.2 today. I updated to salt develop and it now passes another piece of info - the password in the profile does not meet the complexity requirements. I corrected that, and am now able to create machines, but salt-minion doesn't get bootstrapped. So maybe that info about the password complexity will help someone else. v2015.5.2 seemed to swallow that error.

As mentioned, I'm able to create machines, but they never get bootstrapped. salt-cloud hangs indefinitely (or at least for 10+ minutes) here:

[DEBUG   ] Attempting to authenticate as azureuser (try 1 of 15)
[DEBUG   ] SSH command: 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 azureuser@saltbet123.cloudapp.net date'
[DEBUG   ] Child Forked! PID: 30659  STDOUT_FD: 4  STDERR_FD: 7
[DEBUG   ] Terminal Command: /bin/sh -c ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 azureuser@saltbet123.cloudapp.net date
Warning: Permanently added 'saltbet123.cloudapp.net,23.96.11.125' (RSA) to the list of known hosts.
[DEBUG   ] Warning: Permanently added 'saltbet123.cloudapp.net,23.96.11.125' (RSA) to the list of known hosts.
azureuser@saltbet123.cloudapp.net's password: [DEBUG   ] azureuser@saltbet123.cloudapp.net's password:

Permission denied, please try again.
[DEBUG   ] Permission denied, please try again.
azureuser@saltbet123.cloudapp.net's password: [DEBUG   ] azureuser@saltbet123.cloudapp.net's password:

If I ssh manually I'm able to get to the box. I'll keep poking at it.

I guess something we should also be tracking on azure issues is the version of the Azure Python SDK in use. I'm using v0.11.1

rallytime commented 9 years ago

Thanks for the update @bradthurber! I am a little confused about which version you tried. Are you saying you attempted 2015.5.2 AND develop? Or just 2015.5.2?

bradthurber commented 9 years ago

@rallytime I tried both with little success.

I created a new ticket #24968 with details rather than pollute this one with what is likely a different issue

Brad

rallytime commented 9 years ago

@bradthurber Hrm. Even after running into the same provisioning errors you hit in the new ticket you filed, I was still able to delete the VMs successfully, without any errors.

dtoubelis commented 8 years ago

Same issue here:

root@scus-saltmaster:~# salt-cloud --versions-report
Salt Version:
            Salt: 2015.8.1

Dependency Versions:
 Apache Libcloud: 0.15.1
          Jinja2: 2.7.2
        M2Crypto: Not Installed
            Mako: 0.9.1
          PyYAML: 3.10
           PyZMQ: 14.4.0
          Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
            RAET: Not Installed
         Tornado: 4.2.1
             ZMQ: 4.0.4
            cffi: Not Installed
        cherrypy: Not Installed
        dateutil: 1.5
           gitdb: 0.5.4
       gitpython: 0.3.2 RC1
           ioflo: Not Installed
         libnacl: Not Installed
    msgpack-pure: Not Installed
  msgpack-python: 0.3.0
    mysql-python: 1.2.3
       pycparser: Not Installed
        pycrypto: 2.6.1
          pygit2: Not Installed
    python-gnupg: Not Installed
           smmap: 0.8.2
         timelib: Not Installed

System Versions:
            dist: Ubuntu 14.04 trusty
         machine: x86_64
         release: 3.19.0-30-generic
          system: Ubuntu 14.04 trusty

root@scus-saltmaster:~# salt-cloud -f list_nodes azure
[INFO    ] salt-cloud starting
[ERROR   ] There was an error running the function: 'name'
root@scus-saltmaster:~# salt-cloud -f list_nodes azure -l debug
[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Including configuration from '/etc/salt/cloud.conf.d/20_cache.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.conf.d/20_cache.conf
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/mijem.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/mijem.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: scus-saltmaster.scus-saltmaster.j4.internal.cloudapp.net
[DEBUG   ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/azure.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/azure.conf
[DEBUG   ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/azure-servers.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/azure-servers.conf
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/mijem-servers.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/mijem-servers.conf
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Could not LazyLoad parallels.avail_sizes
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy
[DEBUG   ] Could not LazyLoad saltify.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.avail_images
[DEBUG   ] Could not LazyLoad saltify.avail_locations
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] Trying to execute 'azure.list_nodes' with the following kwargs: {}
[ERROR   ] There was an error running the function: 'name'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cloud/cli.py", line 266, in run
    self.function_provider, self.function_name, kwargs
  File "/usr/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1517, in do_function
    driver: self.clouds[fun](call='function')
  File "/usr/lib/python2.7/dist-packages/salt/cloud/clouds/msazure.py", line 226, in list_nodes
    ret[node][prop] = nodes[node][prop]
KeyError: 'name'
root@scus-saltmaster:~#

I do not have any node names that contain dots or begin with number or exceed 15 characters and I'm using azure SDK 0.11.1 (version 1.0.2 does not work).

rallytime commented 8 years ago

@dtoubelis I think this might be a different bug. Would you mind opening a new issue?

dtoubelis commented 8 years ago

@rallytime, submitted as #28150 but I still think it is closely related to this one, only my version of salt is different.

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.