search

splunk / splunk-aws-cloudformation

AWS CloudFormation templates for Splunk distributed cluster deployment
Apache License 2.0
109 stars 66 forks source link

Cloud Formation Fails on us-east-1 t2.micro instances #13

Closed dkindlund closed 7 years ago

dkindlund commented 9 years ago

Loading the master.template on CloudFormation with 2 indexers, I get the following error in the middle of the process:

16:20:06 UTC-0400 CREATE_FAILED AWS::CloudFormation::Stack BastionHost Embedded stack arn:aws:cloudformation:us-east-1:926291082449:stack/Splunk-BastionHost-1JDRJ8V6LBJVQ/189bfc50-5d79-11e5-bea4-50fa1dbb2c64 was not successfully created: The following resource(s) failed to create: [ControllerCondition]. Physical ID:arn:aws:cloudformation:us-east-1:926291082449:stack/Splunk-BastionHost-1JDRJ8V6LBJVQ/189bfc50-5d79-11e5-bea4-50fa1dbb2c64

Can someone let me know how to troubleshoot this further? Thanks!

dkindlund commented 9 years ago

Note: This operation was run using the Admin account.

dkindlund commented 9 years ago

This might be related to issue #5

rarsan commented 9 years ago

Hey @dkindlund, it appears BastionHost EC2 instance bootstrap either failed or timed out. To troubleshoot this further, here are some steps to follow:

  1. In CloudFormation web console, under status tab, refer to the column Status Reason of the resource with logical ID ControllerCondition which failed to create based on the error message above. It should include a specific error message, e.g. "Failed to run cfn-init".
  2. SSH to EC2 instance (BastionHost), and refer to these logs:
    • /var/log/cloud-init.log
    • /var/log/cfn-init.log

Feel free to paste relevant contents from those logs in here. Please note that default behavior of CloudFormation stack creation is to rollback & delete all underlying resources upon failure, so the EC2 instance may have been deleted. You'll need to disable rollback by clicking the right checkbox while going through the CloudFormation create stack wizard.