boto_asg problem if an existing ASG has Target Group attached

[jorotenev]

hi :)

Description of Issue

The root cause of the issue is with the boto library https://github.com/boto/boto/issues/3613 It happens if Autoscaling groups in the AWS account have Target Groups attached to an ASG. The boto's xml parsing fails as the Target Group is not an expected attribute (happens here). I debugged a request to get_all_groups and saw that the XML returned by the raw AWS HTTP API returns correct data but the dict returned by the get_all_groups is malformed - full of None values and invalid structure altogether.

The invalid structure of the dict returned by boto to salt's boto_asg leads to the exception shown below. Since boto seems to be pretty much abandoned a possible solution would be to remove dependencies from boto_asg module to boto.

Setup

generic.sls
{% for dialer_country in countries %}
{% set dialer_name = env.environment  + "-" +  app.name  + "-" + dialer_country -%}
dialer_{{ dialer_country }}:
   boto_asg.present:
    - name: "{{ dialer_name }}"
    - launch_config_name: "{{ dialer_name }}"
    - launch_config:
      - image_id: {{ app.ami }}
      - key_name: {{ env.key_name }}
      - instance_profile_name: arn:aws:iam::{{  env.aws_id }}:instance-profile/{{ app.instance_role }}
      - security_groups:
        - {{ env.security_group_ids.internal_sg }}
        - {{ env.security_group_ids.offices_vpn_sg }}
      - instance_type: t3.micro
      - cloud_init:
         boothooks:
           'boothook.sh': |
             {% set name_ow = {"name_overwrite": app.name + "-" + dialer_country } -%}
             {% set extra_data = {"extra_grains": {"dialer": dialer_country}, "app": name_ow} -%}
             {{ cloud_init_boothook(env, app, extra_data=extra_data) | indent(width=13, indentfirst=False) }}
         scripts:
           'salt_bootstrap.sh': |
             {{ cloud_init_script(env, app, extra_data=extra_data) | indent(width=13, indentfirst=False) }}

    - availability_zones:
      - {{ env.region }}a
      - {{ env.region }}b
    - vpc_zone_identifier:
      - {{ app.vpc_subnets.dialer_1 }}
      - {{ app.vpc_subnets.dialer_2 }}
    - min_size: {{ app.min_size }}
    - max_size: {{ app.max_size }}
    - desired_capacity: {{ app.desired_capacity }}
    - region: {{ env.region }}
    - tags:
      - key: 'country'
        value: {{ dialer_country }}
      - key: 'Name'
        value: {{ dialer_name }}
        propagate_at_launch: True
      - key: 'Environment'
        value: {{ app.tags.env }}
        propagate_at_launch: True
      - key: 'Role'
        value: {{ app.tags.role }}
        propagate_at_launch: True
    - order: last

{% endfor %}

Steps to Reproduce Issue

$ salt "*master*" state.sls dialer.generic
master-10-155.eu-west-1a.qa-public:
----------
          ID: dialer_qa-v2
    Function: boto_asg.present
        Name: qa-public-dialer-qa-v2
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1905, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1830, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/boto_asg.py", line 524, in present
                  asg = __salt__['boto_asg.get_config'](name, region, key, keyid, profile)
                File "/usr/lib/python2.7/dist-packages/salt/modules/boto_asg.py", line 172, in get_config
                  ret[attr] = getattr(asg, attr).split(',')
              AttributeError: 'NoneType' object has no attribute 'split'
     Started: 16:10:08.655691
    Duration: 348.307 ms
     Changes:
----------
          ID: dialer-qa
    Function: boto_asg.present
        Name: qa-public-dialer-qa
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1905, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1830, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/boto_asg.py", line 524, in present
                  asg = __salt__['boto_asg.get_config'](name, region, key, keyid, profile)
                File "/usr/lib/python2.7/dist-packages/salt/modules/boto_asg.py", line 172, in get_config
                  ret[attr] = getattr(asg, attr).split(',')
              AttributeError: 'NoneType' object has no attribute 'split'
     Started: 16:10:09.004262
    Duration: 217.633 ms
     Changes:

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

|  $ salt --versions-report
Salt Version:
           Salt: 2018.3.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: Ubuntu 14.04 trusty
         locale: UTF-8
        machine: x86_64
        release: 3.13.0-116-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.