Open vlebo opened 8 years ago
JensRantil
commented
8 years ago Hi! Which version of SaltStack are you running? Have you checked the Salt master log? Have you made sure that you are seeing events coming in from SQS on your CLI? Maybe https://docs.saltstack.com/en/latest/topics/reactor/#debugging-the-reactor could help you debug master events even more.
vlebo
commented
8 years ago Hi, Salt versionm is: salt-master 2016.3.3 (Boron)
after bumping autoscale this is pretty log
salt/auth { "_stamp": "2016-10-14T17:13:54.723940", "act": "pend", "id": "mlg-gb-api-i-3eb24d0e", "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs52jOA20TyOUi6+DgIv6\nd0EC3x81ffFFzRpn4wVcJRGx2k0HotPlqisoLFVo6xfiCmBCN6fCcItQRNkd33JZ\nHuvfAiAznhkZusSIEroRe6abGkz4E7qUdbSiflzEFQ9k1tlh3JDMqeqEXJx7yXfc\nCsOG6Wwglv9MNu0/lXdTU7lS4T9r+3LWCIm52gjjLXyWysVuY/Me4Q7nlBWBOAtJ\nVWXU4ZNqN4e8+4UV7keyxtsj5fVndwiysunBiJ8/4WheTQprQ/Ha9lXk2ngZWzmT\nkdZDlwQmDl9YPtnJPNr+9xmKNGODQp5aD3F1dy4MVw8sjIJlN+7okZctsheD7sWI\nGwIDAQAB\n-----END PUBLIC KEY-----", "result": true } salt/wheel/20161014171354930099/new { "_stamp": "2016-10-14T17:13:54.932507", "fun": "wheel.key.accept", "fun_args": [], "jid": "20161014171354930099", "user": "Reactor" } salt/key { "_stamp": "2016-10-14T17:13:54.938842", "act": "accept", "id": "mlg-gb-api-i-3eb24d0e", "result": true } salt/wheel/20161014171354930099/ret { "_stamp": "2016-10-14T17:13:54.943430", "fun": "wheel.key.accept", "fun_args": [], "jid": "20161014171354930099", "return": { "minions": [ "mlg-gb-api-i-3eb24d0e" ] }, "success": true, "user": "Reactor" } salt/auth { "_stamp": "2016-10-14T17:14:04.391381", "act": "accept", "id": "mlg-gb-api-i-3eb24d0e", "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs52jOA20TyOUi6+DgIv6\nd0EC3x81ffFFzRpn4wVcJRGx2k0HotPlqisoLFVo6xfiCmBCN6fCcItQRNkd33JZ\nHuvfAiAznhkZusSIEroRe6abGkz4E7qUdbSiflzEFQ9k1tlh3JDMqeqEXJx7yXfc\nCsOG6Wwglv9MNu0/lXdTU7lS4T9r+3LWCIm52gjjLXyWysVuY/Me4Q7nlBWBOAtJ\nVWXU4ZNqN4e8+4UV7keyxtsj5fVndwiysunBiJ8/4WheTQprQ/Ha9lXk2ngZWzmT\nkdZDlwQmDl9YPtnJPNr+9xmKNGODQp5aD3F1dy4MVw8sjIJlN+7okZctsheD7sWI\nGwIDAQAB\n-----END PUBLIC KEY-----", "result": true } minion_start { "_stamp": "2016-10-14T17:14:04.655333", "cmd": "_minion_event", "data": "Minion mlg-gb-api-i-3eb24d0e started at Fri Oct 14 17:14:05 2016", "id": "mlg-gb-api-i-3eb24d0e", "pretag": null, "tag": "minion_start" } salt/minion/mlg-gb-api-i-3eb24d0e/start { "_stamp": "2016-10-14T17:14:04.668006", "cmd": "_minion_event", "data": "Minion mlg-gb-api-i-3eb24d0e started at Fri Oct 14 17:14:05 2016", "id": "mlg-gb-api-i-3eb24d0e", "pretag": null, "tag": "salt/minion/mlg-gb-api-i-3eb24d0e/start" } salt/run/20161014171404949227/new { "_stamp": "2016-10-14T17:14:04.951554", "fun": "runner.autoscaling.highstate_accepted_minion", "fun_args": [], "jid": "20161014171404949227", "user": "Reactor" } salt/run/20161014171404949227/ret { "_stamp": "2016-10-14T17:14:04.992506", "fun": "runner.autoscaling.highstate_accepted_minion", "fun_args": [], "jid": "20161014171404949227", "return": null, "success": true, "user": "Reactor" }
Running salt in debug mode....
[DEBUG ] Rendered data from file: /srv/reactor/autoscaling/minion-connected.sls:
register_minion_if_autoscaling_instance: wheel.key.accept:
also i have this in degub log... maybe this causes issue?
[DEBUG ] Sending event - data = {'fun': 'runner.autoscaling.highstate_accepted_minion', 'fun_args': [], 'jid': '20161014171404949227', 'user': 'Reactor', '_stamp': '2016-10-14T17:14:04.951554'} [DEBUG ] LazyLoaded autoscaling.highstate_accepted_minion Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-08aad64a7b78f19d7 Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-0becdeafaaf051197 Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-06c0816149fdbac94 Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-516e6747 Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-46ca7f50 Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-a1e11091 Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-cadf2efa Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-86833690 Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-d016e7e0 Pending instance registered as EC2 instance, but not as minion: mlg-gb-api-i-4c52e75a The minion is NOT ready for acceptance. [DEBUG ] LazyLoaded nested.output None
newely created instance is not among these
sorry for multiple bumps....
JensRantil
commented
8 years ago I see no events coming in from SQS. Could you please include events also including SQS events to make sure they arriving to the Salt master? Have you made sure to configure an SNS topic that your autoscaling group pushes to? Have you created an SQS queue and made sure it's subscribed to your SNS topic?
Also, please either paste logs like this on gist.github.com or use forward ticks to make the test preformatted. Makes logs a lot readable. See https://guides.github.com/features/mastering-markdown/. Thanks!
vlebo
commented
8 years ago Hi sorry for long and messy reply above :)
I ran trough my stuff again and found an issue which was my fault :)
Everything works now but i still had this error which i fixed by defining db_location:
salt/run/20161017133927841996/ret { "_stamp": "2016-10-17T13:39:27.901987", "fun": "runner.autoscaling.cleanup", "fun_args": [], "jid": "20161017133927841996", "return": "Exception occurred in runner autoscaling.cleanup: Traceback (most recent call last):\n File \"/usr/lib/python2.7/dist-packages/salt/client/mixins.py\", line 356, in low\n data['return'] = self.functions[fun](*args, **kwargs)\n File \"/var/cache/salt/master/extmods/runners/autoscaling.py\", line 84, in cleanup\n registry_call = _registry_call(database_location, registry_script)\nNameError: global name 'database_location' is not defined\n", "success": false, "user": "Reactor" }
Anyway works fine but i still have a question... How long does the process between adding new instance (bump desired by 1) and running highstate takes? I was measuring and it takes between 1 and two minutes for tagging the instance and then 2 more minutes before accepting the key and starting highstate.
Thanks again for everything
V
JensRantil
commented
8 years ago Hi sorry for long and messy reply above :)
No worries ;)
Everything works now but i still had this error which i fixed by defining db_location
Hm, that's odd. By default the runner should use https://github.com/JensRantil/saltstack-autoscaling/blob/master/extensions/runners/autoscaling.py#L12. Where did you define it? Are you running your master as root? Otherwise there could be some file permissions issue. Does the /var/tmp directory exist on your host?
How long does the process between adding new instance (bump desired by 1) and running highstate takes? I was measuring and it takes between 1 and two minutes for tagging the instance and then 2 more minutes before accepting the key and starting highstate.
Something like that, yes. The issue is the installation of the salt-minion. If autoscaling should react faster you could have a look at creating an image with salt-minion already installed and reset cloud-init to set the minion name and start salt-minion. If you do this, would it would be awesome if you could share it back to this project, perhaps on the wiki. I'd be curious to hear how you got it done!
vlebo
commented
8 years ago Hey,
i added it to cleanup() definition
def cleanup(): registry_call = _registry_call(database_location=DEFAULT_DATABASE_LOCATION, registry_script=DEFAULT_REGISTRY_SCRIPT) subprocess.call(registry_call + ['purge', '1', 'days'])
Also i could not use your auto-accept runners in my case because i am using my ASG with CodeDeploy. Unfortunately when using CodeDeploy it starts deploying application at the same time as cloud-init script runs and notifies after it's done so no time for bootstrap before ( needed for code-deploy agent and other packages) so i just accept everything with prefix (for now)... other runners are acting fine. I could maybe solve this with adding lifecycle hooks and put instance in Pendign:Wait until it is finished but still need to figure that out a bit :). For now i have a "check if file exists" in CodeDeploy Install script that waits for bootstrap to finish :) A hack but it works :)
Baking AMI with minion and then setting it's name in cloud-init sounds like smth i could try :) and share here OFC
JensRantil
commented
8 years ago Hm, calls your cleanup function? Are you calling it manually? Or can Saltstack call it automatically somehow?
I'm afraid I've never used CodeDeploy so I can't help with that.
paulrolandw
commented
6 years ago In extensions/runners/autoscaling.py
Should be:
def cleanup(database_location=DEFAULT_DATABASE_LOCATION,registry_script=DEFAULT_REGISTRY_SCRIPT):not
def cleanup():@paulrolandw Would you mind submitting a pull request?
Hi, i am trying to setup this with my ASG but having some issues...
instead of runner.autoscaling.minion_connected in minion-connected.sls and now it works
Any advices?
Thanks!