Tagging lacks good error reporting on failure (was: Setup for authentication...)

[digitaljedi2]

Apologies for all of these 'issues' that not really issues, but I can't find documentation anywhere...

I have a template which validates just fine: bundle exec ruby my-stack.rb validate --parameters "Param1=foo;Param2=bar" --stack-name my-stack --region us-east-1

However when I try to create: bundle exec ruby auvik-cluster.rb create --parameters "Param1=foo;Param2=bar" --stack-name my-stack --region us-east-1 Failed to create stack:

My guess is I am unable to authenticate to cloudformation?

I am able to expand the template and upload it to do the cloudformation run, my question is how are credentials picked up? ENV variables, Profiles/Credentails file, Roles?

[temujin9]

https://github.com/bazaarvoice/cloudformation-ruby-dsl/blob/master/lib/cloudformation-ruby-dsl/cfntemplate.rb#L43 points to http://docs.aws.amazon.com/sdkforruby/api/Aws/CloudFormation/Client.html for how to handle credentials.

[digitaljedi2]

hold on a second here... So as the sdk states... `Credentials

Default credentials are loaded automatically from the following locations:

ENV['AWS_ACCESS_KEY_ID'] and ENV['AWS_SECRET_ACCESS_KEY']`

great let me export credentials...

`jpoole@vadar ~/code/cfn $ export AWS_ACCESS_KEY_ID=myaccess-key

jpoole@home ~/code/cfn/ $ export AWS_SECRET_ACCESS_KEY=mysecret-key`

I know the aws-cli works in the same fashion so a quick test of the credentials:

`jpoole@home ~/code/cfn

$ aws ec2 describe-regions --output=table

| DescribeRegions | +--------------------------------------------------------+ || Regions || |+-----------------------------------+------------------+| || Endpoint | RegionName || |+-----------------------------------+------------------+| || ec2.eu-west-1.amazonaws.com | eu-west-1 || || ec2.ap-southeast-1.amazonaws.com | ap-southeast-1 || || ec2.ap-southeast-2.amazonaws.com | ap-southeast-2 || || ec2.eu-central-1.amazonaws.com | eu-central-1 || || ec2.ap-northeast-1.amazonaws.com | ap-northeast-1 || || ec2.us-east-1.amazonaws.com | us-east-1 || || ec2.sa-east-1.amazonaws.com | sa-east-1 || || ec2.us-west-1.amazonaws.com | us-west-1 || || ec2.us-west-2.amazonaws.com | us-west-2 || |+-----------------------------------+------------------+|`

So credentials work, we should have no problem with calling with the aws-sdk (~>2.2.8) jpoole@home ~/code/cfn $ bundle exec ruby auvik-cluster.rb validate --parameters "Param1=foo;Param2=bar" --stack-name jpoole-test --region us-east-1 Validation successful

Excellent, Template is valid -- and when I try to create... jpoole@home ~/code/cfn $ bundle exec ruby auvik-cluster.rb create --parameters "Param1=foo;Param2=bar" --stack-name jpoole-test --region us-east-1 Failed to create stack:

There is no exception thrown.

If I expand the template and upload it manually, the template works fine. The only thing I can think of is in the template I do need to add CAPABILITY_IAM but not sure if that is handled automatically by the gem or if there is command line options to pass?

Any help would be great.

[temujin9]

CAPABILITY_IAM is handled by the code already. The error you're seeing is just a couple of lines below that; I'm not sure why the error it's wrapping is devoid of useful information, but I suspect its that the AWS Ruby SDK isn't providing a useful error message itself.

I'm not entirely sure of how to help in debugging further. Some things that might help:

1. Run the create command, but throw --verbose after the ruby.
2. Make an incredibly simple stack template and try creating it, to see whether the issue arises from the stack you're making, or the environment you're making it in.
3. Gist or otherwise make visible the template you're using, so we can examine / test it, and see if anything obvious leaps out.

@jonaf Any other suggestions?

[digitaljedi2]

--verbose may show a problem here with the @cfn_client_instance..

/Library/Ruby/Gems/2.0.0/gems/cloudformation-ruby-dsl-1.0.5/lib/cloudformation-ruby-dsl/dsl.rb:24: warning: `&' interpreted as argument prefix /Library/Ruby/Gems/2.0.0/gems/cloudformation-ruby-dsl-1.0.5/lib/cloudformation-ruby-dsl/dsl.rb:93: warning: mismatched indentations at 'end' with 'if' at 90 /Library/Ruby/Gems/2.0.0/gems/aws-sdk-resources-2.2.8/lib/aws-sdk-resources.rb:67: warning: shadowing outer local variable - name /Library/Ruby/Gems/2.0.0/gems/diffy-3.1.0/lib/diffy/diff.rb:43: warning: method redefined; discarding old diff /Library/Ruby/Gems/2.0.0/gems/cloudformation-ruby-dsl-1.0.5/lib/cloudformation-ruby-dsl/cfntemplate.rb:511: warning: method redefined; discarding old exec! /Library/Ruby/Gems/2.0.0/gems/cloudformation-ruby-dsl-1.0.5/lib/cloudformation-ruby-dsl/dsl.rb:67: warning: previous definition of exec! was here /Library/Ruby/Gems/2.0.0/gems/cloudformation-ruby-dsl-1.0.5/lib/cloudformation-ruby-dsl/cfntemplate.rb:42: warning: instance variable @cfn_client_instance not initialized Failed to create stack:

[temujin9]

Hrm. @jonaf, got any quick ideas about what's going on here?

[digitaljedi2]

A smaller stack seems to work just fine... so Perhaps I should gist it up if you have some suggestions on what could the problem be. Possibly too much data? Have to be sent to an s3 bucket first, then processed?

[digitaljedi2]

The file is only 39,472 bytes in size..still under the 51,200 byte limit so surely it's not that I'm sending up too much data in the request, (and likely the appropriate exception would be thrown)... really at a loss here..

jpoole@home ~/code/cfn $ bundle exec ruby jpoole-test.rb expand --parameters "param1=foo;param2=bar" --stack-name jpoole-test --region us-east-1 > jpoole-test.json

jpoole@home ~/code/cfn $ ls -all jpoole-test.json -rw-r--r-- 1 jpoole staff 39472 3 Jan 11:51 jpoole-test.json

[temujin9]

If it's working for some portion but not the whole thing, you can start doing fractional changes, and maybe narrow it down to a specific part of your script that isn't working. As it is, with no code to reference, I can't provide much more help.

[digitaljedi2]

Let me put up a gist...

[temujin9]

It would still be better if you can do the fractional diagnosis yourself. Remove until you have something that works, and then start adding in piecemeal. That will isolate the issue to one or more parts of the code, giving us more information on where to look.

[digitaljedi2]

Ok let me work on that tomorrow and see where it is choking. Appreciate the help thus far.

[digitaljedi2]

So I've stripped this down to as little resources as I can... I can expand and upload the template and things work just fine...
I'm at a loss. But I've committed the stripped down template which really only has a vpc being created.

https://github.com/digitaljedi2/cluster

Appreciate any efforts into debugging this. I'm going to create a new script which will create a cfn client (basically removing the requirement for the cloudformation-ruby-dsl) and see if I can get any different response to pinpoint this further.

[jonaf]

@digitaljedi2 Can you also supply the working, smaller template you mentioned from before?

Reading up the thread, I can't see why creating the stack failed, or why it's not yielding an error. I would try getting closer to the actual error by tracing up the call stack one step at a time. Specifically, see if you can create a new CloudFormation client using the aws-sdk gem and something like:

client = Aws::CloudFormation::Client.new(validate_params: false)

And, of course, actually using the client to, say, list stacks or something.

Once you're sure your client is working, I would proceed with @temujin9 's advice on removing one resource at a time in your template, and retrying, until you get a successful result, and then identify what was the last resource you removed, and we should be able to assist further from there.

[temujin9]

@digitaljedi2 Basically, what @jonaf said, save that I would start from the other direction: start with a mostly empty template (maybe one DNS entry, or something else simple) and add elements piecemeal until it fails. That way, you're sure of the difference when you have added the broken thing (rather than wondering if maybe its failing for different reasons).

[digitaljedi2]

Appreciate the input.

a simple:

!/usr/bin/env ruby

# require 'aws-sdk' require 'yaml'

load credentials from disk

creds = YAML.load(File.read('/Users/jpoole/.aws/credentials.yml'))

client = Aws::CloudFormation::Client.new( validate_params: false, access_key_id: creds['access_key_id'], secret_access_key: creds['secret_access_key'] )

template = File.open("template.json", "rb") contents = template.read resp = client.create_stack({ stack_name: "b-jpoole-auvik-com", # required template_body: contents, parameters: [ { parameter_key: "System", parameter_value: "jpoole.auvik.com", use_previous_value: true, }, { parameter_key: "Cluster", parameter_value: "b", use_previous_value: true, }, ], disable_rollback: true, timeout_in_minutes: 10, capabilities: ["CAPABILITY_IAM"], # accepts CAPABILITY_IAM })

Seems to work just fine...(the template.json was generated with bundle exec ruby cluster.rb expand --parameters "System=jpoole.digitaljedi.ca;Cluster=b" > template.json.

However... $ bundle exec ruby cluster.rb create --parameters "System=jpoole.digitaljedi.ca;Cluster=c" c-jpoole-digitaljedi-ca Failed to create stack:

So I'll try ripping out more out of the template, but vpc's require a minimal set of resources...

[digitaljedi2]

Alright I found the issue... If I remove tag :System => ref('System') tag :Cluster => ref('Cluster') It will work.

[digitaljedi2]

So I guess my question now is, can I reference Parameter values with the _tag :System => ref('System')_ _tag :Cluster => ref('Cluster')_

[temujin9]

Maybe, though that syntax looks funny somehow.

If not, you can reference them with parameters['param_name']. The difference is that the ref is an underlying Cloudformation construct, which can be populated by data from within the engine. Since all you want is the parameter's value, it's not strictly necessary to use.

[temujin9]

Ah, actually: you probably can't use ref there, specifically, because tagging is done differently (and outside of the DSL template entirely).

[temujin9]

It's a parameter passed into the underlying CFN call at a different level, for which ref makes absolutely no sense.

[temujin9]

@jonaf Can we get some clearer error reporting when tagging is bad?

[digitaljedi2]

That did it, Appreciate it.

Apologies for incorrect usage, but would rather an error caught before the cfn.create() executes.

[temujin9]

No apologies necessary; it's a sharp edge. Glad I could help, and hopefully we can get some cleaner error reporting around that fairly easily.