search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.1k stars 5.47k forks source link

Using del_root_vol_on_destroy option in salt-cloud gives IndexError #39257

Closed aig787 closed 7 years ago

aig787 commented 7 years ago

Description of Issue/Question

Using a profile that has del_root_vol_on_destroy: True gives 

'architecture': 'x86_64', 'imageLocation': 'aws-marketplace/CentOS Linux 7 x86_64 HVM EBS 1602-b7ee8a69-ee97-4a49-9e68-afaee216db2e-ami-d7e1d2bd.3', 'rootDeviceType': 'ebs', 'creationDate': '2016-02-26T21:16:29.000Z', 'rootDeviceName': '/dev/sda1', 'imageType': 'machine', 'description': 'CentOS Linux 7 x86_64 HVM EBS 1602'}]'
[INFO    ] Found root device name: /dev/sda1
[ERROR   ] There was a query error: list index out of range
Traceback (most recent call last):
  File "/opt/salt/lib/python2.7/site-packages/salt/cloud/cli.py", line 348, in run
    ret = mapper.run_map(dmap)
  File "/opt/salt/lib/python2.7/site-packages/salt/cloud/__init__.py", line 2231, in run_map
    profile, local_master=local_master
  File "/opt/salt/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1288, in create
    output = self.clouds[func](vm_)
  File "/opt/salt/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 2510, in create
    data, vm_ = request_instance(vm_, location)
  File "/opt/salt/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 1891, in request_instance
    if ex_blockdevicemappings and 'Ebs.VolumeType' not in ex_blockdevicemappings[dev_index]:
IndexError: list index out of range

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: 1.5.0
       cherrypy: 4.0.0
       dateutil: 2.4.2
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: 1.5.0
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: 1.4.4
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Jan 25 2017, 14:57:42)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: 0.6.5
          smmap: 0.9.0
        timelib: 0.2.4
        Tornado: 4.3
            ZMQ: 4.1.2

System Versions:
           dist:
        machine: x86_64
        release: 16.4.0
         system: Darwin
        version: 10.12.3 x86_64
Ch3LL commented 7 years ago

@aig787 i'm not able to replicate this. Can you share a sanitized version of your profile/provider?

also whats the command you are runnign when you see this?

aig787 commented 7 years ago

The command I'm running is salt-cloud -m build/etc/salt/test-cloud.map -c build/etc/salt -l debug -y -P. I am running this on macOS if that makes a difference.

Sanitized profile:

salt-master:
  provider: ec2
  image: ami-d2c924b2
  iam_profile: REDACTED
  size: t2.micro
  ssh_interface: public_ips
  minion:
    environment: qa
  master:
    file_roots:
      qa:
        - /srv/salt
    pillar_roots:
      qa:
        - /srv/pillar/qa
    ext_pillar:
      - file_tree:
          root_dir: /srv/pillar
  location: us-west-2
  availability_zone: us-west-2c
  ssh_username: centos
  tag: {'service': 'salt'}
  subnetid: REDACTED
  del_root_vol_on_destroy: True
  securitygroupid:
    - REDACTED

Sanitized provider:

ec2:
  driver: ec2
  ssh_interface: private_ips
  keyname: REDACTED
  private_key: REDACTED
  securitygroupid: REDACTED
  id: REDACTED
  key: REDACTED
  location: us-west-2
Ch3LL commented 7 years ago

@aig787 i'm still not able to replicate this. I wonder if its related to the drives attached to that ami.

Can you add the following and paste here with it sanitized:

             # Use default volume type if not specified
+            log.debug('blockdevicemappings are the following: {0}'.format(ex_blockdevicemappings))
+            log.debug('dev_index is the following:'.format(dev_index))
             if ex_blockdevicemappings and 'Ebs.VolumeType' not in ex_blockdevicemappings[dev_index]:
                 type_key = '{0}BlockDeviceMapping.{1}.Ebs.VolumeType'.format(spot_prefix, dev_index)
                 params[type_key] = rd_type

I'm curious what values you are getting there.

aig787 commented 7 years ago

Adding those lines I now get this output

[INFO    ] Found root device name: /dev/sda1
[DEBUG   ] blockdevicemappings are the following: None
[DEBUG   ] dev_index is the following:
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
tedohayer commented 7 years ago

Just ran into this. My cloud profile entry:

 ec2.oregon.medium:
  del_all_vol_on_destroy: true
  del_root_vol_on_destroy: true
  image: ami-d2c924b2
  network_interfaces:
  - DeviceIndex: 0
    SecurityGroupId:
    - sg-foobar
    SubnetId: subnet-foobar
  provider: ec2-or
  size: m3.medium
  ssh_username: centos
  sync_after_install: grains

My cloud map entry:

ec2.oregon.medium:
- web03.or.company.internal:
    grains:
      branch: master
      datacenter: or.company.internal
      role:
      - prod
      - web
      salt_managed: true
    minion:
      master: salt01.or.company.internal
    network_interfaces:
    - AssociatePublicIpAddress: false
      DeviceIndex: 0
      SecurityGroupId:
      - sg-foobar
      SubnetId: subnet-foobar
    salt_interface: private_ips
    block_device_mappings:
    - DeviceName: /dev/sda1
      Ebs.VolumeSize: 30

Removing del_root_vol_on_destroy from the profile allowed salt to build the instance, I was able to keep del_all_vol_on_destroy.

Ch3LL commented 7 years ago

@aig787 thnaks for adding those logs. I don't quit understand why you are hitting that indexerror if ex_blockdevicemappings is None, so we will need to get an engineer on this issue. Thanks for all your help on this one. Also thanks @tedohayer for the profile information. We will definitely need to get htis fixed.

rallytime commented 7 years ago

@aig787 I have been looking at this today and I am unable to reproduce this either. Would you mind trying this again with those same debug lines that @Ch3LL asked you about above, and then adding a couple of additional type check debug lines? I am wondering if somehow the ex_blockdevicemappings value is getting set to a 'None' string and that's how it's getting past the if ex_blockdevicemappings check.

I also noticed that the first time we had these debug lines in there, we were missing the {0} from the string format. So that's why nothing printed on that second debug statement after the :. I'm curious what index is listed there as well.

             # Use default volume type if not specified
+            log.debug('blockdevicemappings are the following: {0}'.format(ex_blockdevicemappings))
+            log.debug('blockdevcemappings type: {0}'.format(type(ex_blockdevicemappings)))
+            log.debug('dev_index is the following: {0}'.format(dev_index))
+            log.debug('dev_index type: {0}'.format(type(dev_index)))
             if ex_blockdevicemappings and 'Ebs.VolumeType' not in ex_blockdevicemappings[dev_index]:
                 type_key = '{0}BlockDeviceMapping.{1}.Ebs.VolumeType'.format(spot_prefix, dev_index)
                 params[type_key] = rd_type