search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.22k stars 5.49k forks source link

salt-cloud vsphere cloning and creating vm's #17061

Closed ingwa closed 9 years ago

ingwa commented 10 years ago

Hi there,

I've finally managed to get salt cloud talking correctly to esxi and vsphere in general. Turns out to clone images or create vm's on esxi, you MUST have vsphere center server or similar orchestration management system from them in place to expose those elements of the api. The esxi api does not have that in place.

Having issues now on machine creation, seems there is a string int mixup somewhere: 

[DEBUG   ] clone_kwargs are set to {'datastore': None,
 'folder': None,
 'host': None,
 'name': 'newlinux',
 'resourcepool': 'resgroup-11',
 'template': False}
[ERROR   ] There was a profile error: string indices must be integers, not str
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/cli.py", line 239, in run
    self.config.get('names')
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1364, in run_profile
    ret[name] = self.create(vm_)
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/__init__.py", line 1234, in create
    output = self.clouds[func](vm_)
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/clouds/vsphere.py", line 248, in create
    deploy_kwargs = _deploy(vm_)
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/clouds/vsphere.py", line 308, in _deploy
    ip_address = wait_for_ip(vm_['name'])
  File "/usr/local/lib/python2.7/dist-packages/salt/cloud/clouds/vsphere.py", line 293, in wait_for_ip
    log.debug('Pulling VM {0} {1} seconds for an IP address'.format(vm_['name']))
TypeError: string indices must be integers, not str

Here is my profile: 

my-ubuntu-vm:
  provider: my-provider
  image: ubuntu
  minion:
    master: 192.168.1.15
  ssh_username: userabc
  ssh_password: passxyz
  script_args: -c /tmp/
  resourcepool: resgroup-11

I called it like this:

salt-cloud -p my-ubuntu-vm newlinux -l debug

Any ideas on what I'm doing wrong?

steverweber commented 10 years ago

looks like I caused this issue. I'll take a look.

steverweber commented 10 years ago

https://github.com/saltstack/salt/pull/17071

steverweber commented 10 years ago

to work around this you can set "deploy: False"

        mfcf_centos6:
            provider: mfcf
            image: _centos_6_5
            resourcepool: resgroup-41098
            folder: TEST
            deploy: False
rallytime commented 10 years ago

@steverweber Thanks for the quick PR!

@ingwa Can you give this fix a try when you have a moment? I am also backporting the fix to the 2014.7 branch because it looks like that code was changed there as well.

rallytime commented 10 years ago

Actually, I was looking at the wrong branch. Looks like this bug is only on develop, so I won't be backporting the PR. My mistake!

ingwa commented 10 years ago

@steverweber Hi Steve, I did that and the vm successfully created with a nice output of all the fields. Only thing is salt doesn't appear to have installed.

Versions: 

salt-master 2014.7.0-911-ge243a65 (Helium)
salt-minion 2014.7.0-911-ge243a65 (Helium)
salt-cloud 2014.7.0-911-ge243a65 (Helium)
steverweber commented 10 years ago

/ salt is not installed on the new system because deploy: False /

if you reinstall salt using the latest develop branch it will have the fix so you can use deploy: True.

Or you could manualy update your salt code with this fix. https://github.com/steverweber/salt/commit/f8556451e517953cb980623fc680e79332ac2a19

I havent had a chance to test Deploy: True, please let me know if you get it to work. Things that the documents are likly missing is that VM tools will need to be installed on the template/vm you clone. Also after the new instance boots for the first time, it must have an ip address or get one from dhcp.

mochrul commented 10 years ago

It's not related to the original question, but the vsphere provider does not use the ssh_username and the ssh_password keywords. It uses the template_user(name)? and template_password keywords. At least it not worked for me to use ssh_username. And even there were no default as in any other provider.

mochrul commented 10 years ago

Seeing the code, isn't a patch like this also needed to make it run? I'm really just curious.

diff --git a/salt/cloud/clouds/vsphere.py b/salt/cloud/clouds/vsphere.py index e3082c2..eb58818 100644 --- a/salt/cloud/clouds/vsphere.py +++ b/salt/cloud/clouds/vsphere.py @@ -305,7 +305,7 @@ def deploy(vm): '''

TODO: review salt.utils.cloud.bootstrap(vm_, opts)

 # TODO: review salt.utils.cloud.wait_for_ip
 template_user = config.get_cloud_config_value(
     'template_user', vm_, __opts__
steverweber commented 10 years ago

Thanks, I missed that.

ingwa commented 10 years ago

Hi guys, appreciate all your input!

@steverweber thanks for making the updates, seems we're on the way there. My latest version is:

salt-cloud 2014.7.0-990-g023e835 (Helium)
salt-master 2014.7.0-990-g023e835 (Helium)
salt-minion 2014.7.0-990-g023e835 (Helium)

@mochrul thanks for your input re the passwords.

I now have this in my ubuntu.conf profile:

salt-cloud 2014.7.0-990-g023e835 (Helium)
salt-master 2014.7.0-990-g023e835 (Helium)
salt-minion 2014.7.0-990-g023e835 (Helium)
root@salt:/etc/salt/cloud.profiles.d# vim ubuntu.conf 

ubuntu:
  provider: my-provider
  image: ubuntu
  template_username: "redacted"
  template_password: "redacted"
  ssh_username: "redacted"
  ssh_password: "redacted"
  script_args: -c /tmp/
  resourcepool: resgroup-11
  deploy: True

When I run the create function, it creates successfully, machine spins up and then it tries to connect. I'm getting this: 

[DEBUG   ] Deploying 192.168.1.22 at 1415140802.0
[DEBUG   ] Attempting connection to host 192.168.1.22 on port 22
[DEBUG   ] SSH port 22 on 192.168.1.22 is available
[DEBUG   ] Using password authentication
[DEBUG   ] Attempting to authenticate as None (try 1 of 15)
[DEBUG   ] SSH command: 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 None@192.168.1.22 date'
[DEBUG   ] Child Forked! PID: 25399  STDOUT_FD: 6  STDERR_FD: 8
[DEBUG   ] Terminal Command: /bin/sh -c ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 None@192.168.1.22 date
Warning: Permanently added '192.168.1.22' (ECDSA) to the list of known hosts.
[DEBUG   ] Warning: Permanently added '192.168.1.22' (ECDSA) to the list of known hosts.
None@192.168.1.22's password: [DEBUG   ] None@192.168.1.22's password:

Permission denied, please try again.
[DEBUG   ] Permission denied, please try again.
None@192.168.1.22's password: [DEBUG   ] None@192.168.1.22's password:

Permission denied, please try again.
[DEBUG   ] Permission denied, please try again.
None@192.168.1.22's password: [DEBUG   ] None@192.168.1.22's password:

Permission denied (publickey,password).
[DEBUG   ] Permission denied (publickey,password).
[DEBUG   ] Using password authentication
[DEBUG   ] Attempting to authenticate as None (try 2 of 15)

*** Truncated***

Anything I've done?

(corrected spelling in config)

ingwa commented 10 years ago

FYI I have also tried with just user: and password: in the config however the connection debug always states it's using "None".

ingwa commented 10 years ago

Ok, I did some more testing and it seems the template_user: is the correct format for the username, however for some reason the password won't get taken. Here's the output from the debug: 

[DEBUG   ] Deploying 192.168.1.4 at 1415142853.0
[DEBUG   ] Attempting connection to host 192.168.1.4 on port 22
[DEBUG   ] SSH port 22 on 192.168.1.4 is available
[DEBUG   ] Using password authentication
[DEBUG   ] Attempting to authenticate as redacteduser (try 1 of 15)
[DEBUG   ] SSH command: 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 redacteduser@192.168.1.4 date'
[DEBUG   ] Child Forked! PID: 26509  STDOUT_FD: 6  STDERR_FD: 8
[DEBUG   ] Terminal Command: /bin/sh -c ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 redacteduser@192.168.1.4 date
Warning: Permanently added '192.168.1.4' (ECDSA) to the list of known hosts.
[DEBUG   ] Warning: Permanently added '192.168.1.4' (ECDSA) to the list of known hosts.
redacteduser@192.168.1.4's password: [DEBUG   ] redacteduser@192.168.1.4's password:

Tue Nov  4 15:14:50 PST 2014
[DEBUG   ] Tue Nov  4 15:14:50 PST 2014
[DEBUG   ] Logging into 192.168.1.4:22 as redacteduser
[DEBUG   ] SSH command: 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 redacteduser@192.168.1.4 \'test -e \\"/tmp/.saltcloud-1f679a23-3da4-4abd-a559-3d7673c161eb\\"\''
[DEBUG   ] Child Forked! PID: 26511  STDOUT_FD: 6  STDERR_FD: 8
[DEBUG   ] Terminal Command: /bin/sh -c ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -p 22 redacteduser@192.168.1.4 'test -e \"/tmp/.saltcloud-1f679a23-3da4-4abd-a559-3d7673c161eb\"'
Warning: Permanently added '192.168.1.4' (ECDSA) to the list of known hosts.
[DEBUG   ] Warning: Permanently added '192.168.1.4' (ECDSA) to the list of known hosts.
redacteduser@192.168.1.4's password: [DEBUG   ] redacteduser@192.168.1.4's password:
Usage: salt-cloud

salt-cloud: error: There was a profile error: A wrong password has been issued while establishing ssh session

I have successfully logged into the ssh client after the machine spun up manually using the same credentials in the input file. I trimmed it up too by removing ssh_username, etc.

steverweber commented 10 years ago

thanks for testing this. I think I'll take a look tomorrow.

steverweber commented 10 years ago

looking at the source... this wont work... the code will need to be changed :(

I might take the time to migrate the code to use: salt.utils.cloud.bootstrap

or pass 'has_ssh_agent' to the deploy method so the password will take...

Note to self:
salt / salt / utils / cloud.py

    has_ssh_agent = False
    if opts.get('ssh_agent', False) and 'SSH_AUTH_SOCK' in os.environ:
        if stat.S_ISSOCK(os.stat(os.environ['SSH_AUTH_SOCK']).st_mode):
            has_ssh_agent = True

912: def deploy_script(host,
...
            elif password and 'has_ssh_agent' in kwargs and kwargs['has_ssh_agent'] is False:
                log.debug('Using {0} as the password'.format(password))
                ssh_kwargs['password'] = password
ingwa commented 10 years ago

@steverweber appreciate you looking into this. Once it's all up and running I'll try work on a detailed example for the documentation.

steverweber commented 10 years ago

sounds good. I'll clean up the code a little so it can take in:

ssh_username
ssh_password

I'm tempted to deprecat http://docs.saltstack.com/en/latest/topics/development/deprecations.html

template_user
template_password
techhat commented 10 years ago

I would support deprecating template_user and template_password.

ingwa commented 10 years ago

Something else that I observed yesterday with all my tests. I notice that part of the command to create a new vm is to define it's name, salt-cloud -p profilename newname. Newname is then used to create a pre-authorized key for salt-key, however when the template vm spins up it's host name is generic, and salt then registers the old hostname from the vm, and places it in unapproved. Secondly, even though I defined:

minion:
  master: mymasterip

I tried setting this in the profile, provider, and cloud configs however nothing would allow overriding the default "salt" local network name.

My next bet is to perhaps write my own startup sh script that pulls the bootstrap, then does a rename on the hostname to get the new name for the vm and use that instead, and then restart salt, but I am wondering isn't this done somehow already?

ingwa commented 10 years ago

FYI, the tests I performed were after applying the hot fix directly to my salt code and running it again. It did work with the overrides in place, but really appreciate you all looking into this to get it working. I have a few cloud systems I'm testing but one step at a time hehe :)

rallytime commented 10 years ago

@steverweber I see you have a commit referencing this issue, but it doesn't appear to be related to a pull request. Are you still working on that, or did it just get missed?

steverweber commented 10 years ago

Sorry, Its been one of them weeks... Had no time. I can say the second week of December I'll have resolve this. On Nov 18, 2014 6:26 PM, "Nicole Thomas" notifications@github.com wrote:

@steverweber https://github.com/steverweber I see you have a commit referencing this issue, but it doesn't appear to be related to a pull request. Are you still working on that, or did it just get missed?

— Reply to this email directly or view it on GitHub https://github.com/saltstack/salt/issues/17061#issuecomment-63565492.

rallytime commented 10 years ago

Not a problem at all @steverweber! I just wanted to check in. Thanks again for all of your contributions!

steverweber commented 9 years ago

ok i bookmarked this issue and will give it a go next week. thanks for waiting.

rallytime commented 9 years ago

The vsphere driver is on a deprecation path and will be officially removed in Salt Carbon (see #24487 and #24504). As such, we'd recommend moving to the new vmware driver. Since we have a much more robust and clean driver, we won't be spending time cleaning up bugs for vsphere. However, we will of course be accepting fixes from community members as they are submitted.

nmadhok commented 9 years ago

@rallytime I think this issue can be closed now.

steverweber commented 9 years ago

cheers!

molixrawdi commented 9 years ago

Hi, Thanks for all your efforts, may I just confirm that users of SaltStack can expect to: 1- Create a virtual machine via template file, using salt-cloud/pyVmomi, via vcenter only as vsphere is deprecated. 2- Clone an existing virtual machine, using salt-cloud/pyVmomi, via vcenter only as vsphere is deprecated.

Best Regards M

molixrawdi commented 9 years ago

Could you please confirm that this link is deprecated: https://docs.saltstack.com/en/latest/ref/clouds/all/salt.cloud.clouds.vsphere.html

rallytime commented 9 years ago

@molixrawdi Correct - the vsphere driver is officially on a deprecation path and will be removed from Salt in the Carbon release. The VMWare driver in Salt-Cloud should be used instead:

molixrawdi commented 9 years ago

Hi @rallytime many thanks for the update, Am I correct about the two points I made earlier? (1-Create Virtual machine, 2- Clone virtual machine)? By mentioning the vmware drivers as replacement, Am I correct in assuming by this you mean 'pyVmomi' = https://pypi.python.org/pypi/pyvmomi

Best Regards Molix

rallytime commented 9 years ago

@molixrawdi - There is a salt-cloud driver named vmware.py in the salt/cloud/clouds/ directory of Salt, which is the salt-cloud driver I was referring to above. The docs that I linked in my earlier comment explain how to use that driver. The salt-cloud vmware.py driver depends on the pyVmomi package installable through pip.

Here's the salt-cloud driver I am referring to: https://github.com/saltstack/salt/blob/2015.8/salt/cloud/clouds/vmware.py

Does that help clear things up?

molixrawdi commented 9 years ago

@rallytime Thanks again, so the vmware.py comes with salt-cloud which I have so far managed to install using pip install salt-cloud? (Sorry for asking many questions and I truly appreciate your help and time), finally about my two assumptions are they correct!

They are:

1- Create a virtual machine via template file, using salt-cloud/pyVmomi, via vcenter only as vsphere is deprecated. 2- Clone an existing virtual machine, using salt-cloud/pyVmomi, via vcenter only as vsphere is deprecated.

rallytime commented 9 years ago

@molixrawdi Yes, the vmware.py driver comes with salt-cloud. That driver is how salt-cloud interacts with/connects to your VMWare installation. How did you install salt? Depending on your operating system, salt-cloud is either included by default with your salt installation, or it is an extra package taht you can install. You could use pip as well if you want, but it might already be available. Also, what version of salt are you running? (Please post the output of salt-cloud --versions-report.) This will determine if you have the correct vmware.py driver available by default in your salt installation.

I guess I don't understand what you're stating with your assumptions. You can create and clone a virtual machine with a template file with the vmware driver that comes with salt-cloud. But before you do that you need to install pyVmomi, because the salt-cloud vmware.py driver uses the pyVmomi driver.

From the the top of the vmware driver docs: "The VMware cloud module allows you to manage VMware ESX, ESXi, and vCenter."

molixrawdi commented 9 years ago

Hi @rallytime I have run the command, please see this output below:

salt01 ~]# salt-cloud --versions-report
Salt Version:
            Salt: 2015.8.0

Dependency Versions:
 Apache Libcloud: 0.18.0
          Jinja2: 2.2.1
        M2Crypto: Not Installed
            Mako: Not Installed
          PyYAML: 3.11
           PyZMQ: 14.5.0
          Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
            RAET: Not Installed
         Tornado: 4.2.1
             ZMQ: 4.0.5
            cffi: Not Installed
        cherrypy: Not Installed
        dateutil: 1.4.1
           gitdb: Not Installed
       gitpython: Not Installed
           ioflo: Not Installed
         libnacl: Not Installed
    msgpack-pure: Not Installed
  msgpack-python: 0.4.6
    mysql-python: Not Installed
       pycparser: Not Installed
        pycrypto: 2.6.1
          pygit2: Not Installed
    python-gnupg: Not Installed
           smmap: Not Installed
         timelib: Not Installed

System Versions:
            dist: centos 6.7 Final
         machine: x86_64
         release: 2.6.32-573.7.1.el6.x86_64
          system: CentOS 6.7 Final
molixrawdi commented 9 years ago

I have also a second node for testing running Ubuntu 14.04:

salt02:~$ salt-cloud --versions-report
                  Salt: 2015.5.3
                Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
                Jinja2: 2.7.2
              M2Crypto: 0.21.1
        msgpack-python: 0.3.0
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: Not Installed
               Tornado: Not Installed
       Apache Libcloud: 0.18.0
 Debian source package: 2015.5.3+ds-1trusty1
molixrawdi commented 9 years ago

I have tested both with multiple commands from salt to ensure that there are no outstanding compatibility issues with libraries or clashes with versions.

molixrawdi commented 9 years ago

After testing, I came to the conclusion that one must install 'salt-cloud' via 'apt-get' and the salt repository instead of 'pip install' the pip install will fail due to unsolved dependencies.

rallytime commented 9 years ago

@molixrawdi Ok - So the vmware.py driver isn't included by default in the 2015.5.3 release as it was added later to the 2015.5 branch. (It's there in the 2015.8.0 release though.) If you want to use the vmware.py driver in your 2015.5.3 install, you need to grab the driver and put it in your clouds directory. Since you're on Ubuntu, this should look something like /usr/lib/python2.7/dist-packages/salt/cloud/clouds/. Then you should be able to follow the docs to get your Salt Cloud configuration files set up.

There's a ton of useful information about Salt Cloud in the Salt Cloud Docs.