search

saltstack-formulas / ec2-autoscale-reactor

Autonomous Minion Management via EC2 Autoscaler
http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html
Other
77 stars 28 forks source link

Failed to Render #2

Closed rawkode closed 10 years ago

rawkode commented 10 years ago

I've been trying to configure this to work with my auto-scaling groups, but I can't seem to see past the following error:

2014-08-11 19:48:17,101 [salt.utils.event ][ERROR ] Failed to render "/srv/reactor/ec2-autoscale.sls"

techhat commented 10 years ago

Thanks for the report @rawkode, can you give me the relevant and sanitized configs for this, and the output from salt --versions-report? I assume you're running the develop or 2014.7 branch, but the rest of the info will be helpful.

rawkode commented 10 years ago 
root@ip-172-31-160-84:/srv/reactor# salt --versions-report
           Salt: 2014.1.10
         Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
 msgpack-python: 0.3.0
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
         PyYAML: 3.10
          PyZMQ: 14.0.1
            ZMQ: 4.0.4

I've got debug enabled for cherrypy, but nothing else seems to show up in the logs.

reactor:
  - 'salt/netapi/hook/ec2/autoscale':
    - '/srv/reactor/ec2-autoscale.sls'

rest_cherrypy:
  port: 8080
  debug: True
  host: 0.0.0.0
  disable_ssl: True
  webhook_url: /hook
  webhook_disable_auth: True

If there's anything else I can provide, just let me know.

techhat commented 10 years ago

Just to verify, are you running 2014.1.10, or are you running out of the develop or 2014.7 branch and it's just showing up wrong?

rawkode commented 10 years ago

I added the ppa with

sudo add-apt-repository ppa:saltstack/salt

I thought this would get the latest release, 2014.7 - is that not correct?

techhat commented 10 years ago

It would, were 2014.7 an official release. But I don't think it's even hit RC stage yet. @basepi, do we know how far off that is?

In the meantime, you'll want to go with a git checkout. If you check out the salt-bootstrap project, it can install from a specific git branch for you.

rawkode commented 10 years ago

Ah, right. No problem. I'll check out salt-bootstrap.

Thanks for your help

basepi commented 10 years ago

The release candidate is cut and the tag has been pushed.

Here's the release on PyPI: https://pypi.python.org/pypi?:action=display&name=salt&version=2014.7.0rc1

And here's the tag: https://github.com/saltstack/salt/tree/v2014.7.0rc1

rawkode commented 10 years ago

Got it running using salt-bootstrap.

Thanks for the help

rawkode commented 10 years ago

Edit: Nevermind - it took a while, but we're playing the same game now. Python is tough, from a PHP angle. Anyway, forwards! :)

techhat commented 10 years ago

Sorry it took so long to respond here @rawkode, glad you got it working!

rawkode commented 10 years ago

Hi @techhat,

I've asked on IRC, but not had a reply yet. salt-master is accepting the keys now, which is great. I salt-cloud doesn't seem to bootstrap the machine though. Is there some salt-cloud configuration I'll need?

I've been reading the salt-cloud docs and I've seen stuff about profiles, but not sure if I'm on the right path.

techhat commented 10 years ago

This can be a tricky one, because EC2 offers so many ways to handle things. First off, are your master and minions both in the same VPC in EC2? If they're on the same subnet, the master needs to use the minion's private IP, which means ssh_interface needs to be set to private_ips (the default is public_ips).

For further troubleshooting: since the master process is what is kicking off everything, run it in the foreground with -l debug so that you can watch salt.cloud work. It might give you additional insight as to what is working and what is failing.

rawkode commented 10 years ago

Yeah, I've got that setting enabled. I've attached the debug output, but it doesn't seem to offer much information ... at least to me.

ip-172-16-224-114 is the salt-master, which seems to find itself, though salt-minion isn't installed locally.

Then it accepts the key for the autoscaled box, but nothing seems to happen next. I've attempted to ssh to the autoscaled box using the key and username defined in the ec2 provider, and everything seems fine.

[DEBUG   ] Gathering reactors for tag salt/netapi/hook/ec2/autoscale
[DEBUG   ] Compiling reactions for tag salt/netapi/hook/ec2/autoscale
[INFO    ] Starting new HTTPS connection (1): sns.eu-west-1.amazonaws.com
[DEBUG   ] "GET /SimpleNotificationService-e372f8ca30337fdb084e8ac449342c77.pem HTTP/1.1" 200 1854
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'id': 'i-9c3356df', '_stamp': '2014-08-15T14:41:18.190425', 'result': True, 'act': 'delete'}
[DEBUG   ] Gathering reactors for tag salt/key
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Guessing ID. The id can be explicitly in set /etc/salt/minion
[INFO    ] Found minion id from generate_minion_id(): ip-172-16-224-114
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/ec2-autoscale-reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/ec2-autoscale-reactor.conf
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] LazyLoaded local_cache.clean_old_jobs
[DEBUG   ] This salt-master instance has accepted 0 minion keys.
[DEBUG   ] Gathering reactors for tag salt/netapi/hook/ec2/autoscale
[DEBUG   ] Compiling reactions for tag salt/netapi/hook/ec2/autoscale
[INFO    ] Starting new HTTPS connection (1): sns.eu-west-1.amazonaws.com
[DEBUG   ] "GET /SimpleNotificationService-e372f8ca30337fdb084e8ac449342c77.pem HTTP/1.1" 200 1854
[ERROR   ] Failed to render "/srv/reactor/ec2-autoscale.sls"
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Guessing ID. The id can be explicitly in set /etc/salt/minion
[INFO    ] Found minion id from generate_minion_id(): ip-172-16-224-114
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/ec2-autoscale-reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/ec2-autoscale-reactor.conf
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] LazyLoaded local_cache.clean_old_jobs
[DEBUG   ] This salt-master instance has accepted 0 minion keys.
[DEBUG   ] Gathering reactors for tag salt/netapi/hook/ec2/autoscale
[DEBUG   ] Compiling reactions for tag salt/netapi/hook/ec2/autoscale
[INFO    ] Starting new HTTPS connection (1): sns.eu-west-1.amazonaws.com
[DEBUG   ] "GET /SimpleNotificationService-e372f8ca30337fdb084e8ac449342c77.pem HTTP/1.1" 200 1854
[DEBUG   ] Gathering reactors for tag salt/run/20140815144312328384/new
[DEBUG   ] Gathering reactors for tag salt/cloud/i-b1086df2/creating
[DEBUG   ] Gathering reactors for tag salt/run/20140815144312328384/ret
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Guessing ID. The id can be explicitly in set /etc/salt/minion
[INFO    ] Found minion id from generate_minion_id(): ip-172-16-224-114
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/ec2-autoscale-reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/ec2-autoscale-reactor.conf
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] LazyLoaded local_cache.clean_old_jobs
[DEBUG   ] This salt-master instance has accepted 1 minion keys.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Guessing ID. The id can be explicitly in set /etc/salt/minion
[INFO    ] Found minion id from generate_minion_id(): ip-172-16-224-114
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/ec2-autoscale-reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/ec2-autoscale-reactor.conf
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] LazyLoaded local_cache.clean_old_jobs
[DEBUG   ] This salt-master instance has accepted 1 minion keys.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Guessing ID. The id can be explicitly in set /etc/salt/minion
[INFO    ] Found minion id from generate_minion_id(): ip-172-16-224-114
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/ec2-autoscale-reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/ec2-autoscale-reactor.conf
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] LazyLoaded local_cache.clean_old_jobs
[DEBUG   ] This salt-master instance has accepted 1 minion keys
rawkode commented 10 years ago

Hi @techhat,

There seems to be a bug in salt, which I've gotten around (Though I'm not convinced in the correct way, so I won't submit a pull-request without your assistance)

I found the following code inside cloud/clouds/ec2.py

    if 'instance_id' in vm_:
        # This was probably created via another process, and doesn't have
        # things like salt keys created yet, so let's create them now.
        if 'pub_key' not in vm_ and 'priv_key' not in vm_:
            log.debug('Generating minion keys for {0[name]!r}'.format(vm_))
            vm_['priv_key'], vm_['pub_key'] = salt.utils.cloud.gen_keys(
                salt.config.get_cloud_config_value(
                    'keysize',
                    vm_,
                    __opts__
                )
            )
    else:
        # Put together all of the information required to request the instance,
        # and then fire off the request for it
        data, vm_ = request_instance(vm_, location)

        # Pull the instance ID, valid for both spot and normal instances

        # Multiple instances may have been spun up, get all their IDs
        vm_['instance_id_list'] = []
        for instance in data:
            vm_['instance_id_list'].append(instance['instanceId'])

    vm_['instance_id'] = vm_['instance_id_list'].pop()
    if len(vm_['instance_id_list']) > 0:
        # Multiple instances were spun up, get one now, and queue the rest
        queue_instances(vm_['instance_id_list'])

    # Wait for vital information, such as IP addresses, to be available
    # for the new instance
    data = query_instance(vm_)

It would seem that these three lines:

    vm_['instance_id'] = vm_['instance_id_list'].pop()
    if len(vm_['instance_id_list']) > 0:
        # Multiple instances were spun up, get one now, and queue the rest
        queue_instances(vm_['instance_id_list'])

should be indented, to be within the else statement.

I'm now fully working :+1:

techhat commented 10 years ago

Thanks for grabbing that @rawkode! I've merged in your pull request, so you should be good to go.