Miasma-aws invalid STS token errors when using IAM instance profile.

[petecheslock]

We were previously using v0.1.26 of miasma-aws with chef-server-populator. I ran into this issue.

NoMethodError: undefined method `with_headers' for #<HTTP::Client:0x00000001e73050>
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/miasma-aws-0.1.26/lib/miasma/contrib/aws.rb:578:in `connection'

I started the upgrade process and found the following between versions 0.1.26 and 0.1.28

remote = Miasma.api(:provider => 'aws', :type => 'storage', :credentials => {:aws_iam_instance_profile => true})
=> <Miasma::Models::Storage::Aws:35557620 [{}]>

In 0.1.28

remote = Miasma.api(:provider => 'aws', :type => 'storage', :credentials => {:aws_iam_instance_profile => true})
=> <Miasma::Models::Storage::Aws:101162920 [{"aws_sts_token_expires"=>2016-01-10 09:23:55 UTC}]>

I went thru upgrading to v0.1.36 to grab this https://github.com/miasma-rb/miasma-aws/commit/98973a30044186e3f3271692bae022693eb017ec - and now i get the following.

[2] pry(main)> remote = Miasma.api(:provider => 'aws', :type => 'storage', :credentials => {:aws_iam_instance_profile => true})
=> <Miasma::Models::Storage::Aws:28839540 [{"aws_sts_token_expires"=>2016-01-10 09:23:55 UTC}]>
[3] pry(main)> remote.buckets.get('my-bucket')
Miasma::Error::ApiError::RequestError: Forbidden - InvalidClientTokenId: The security token included in the request is invalid.
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/miasma-0.2.36/lib/miasma/types/api.rb:124:in `block in request'

Same issue in 0.2.0 as well.

Let me know if you need any other info.

[dougireton]

I get the same error.

.sfn file

# ~/.sfn
Configuration.new do
  processing true
  credentials do
    provider :aws
    aws_region 'us-west-2'
    aws_profile_name 'my-custom-profile'
  end
end

.aws/credentials file

# ~/.aws/credentials
[my-custom-profile]
aws_access_key_id=<MY_VALID_KEY>
aws_secret_access_key=<my_secret_access_key>
aws_session_token="my-session-token"
aws_security_token="my-session-token"

sfn list

$ DEBUG=1 chef exec sfn list

ERROR: Miasma::Error::ApiError::RequestError: Forbidden
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-0.2.38/lib/miasma/types/api.rb:124:in `block in request'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-0.1.32/lib/bogo/retry.rb:65:in `call'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-0.1.32/lib/bogo/retry.rb:65:in `run!'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-0.2.38/lib/miasma/types/api.rb:146:in `retryable_request'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-0.2.38/lib/miasma/types/api.rb:121:in `request'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-aws-0.2.0/lib/miasma/contrib/aws/orchestration.rb:74:in `block in load_stack_data'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-aws-0.2.0/lib/miasma/contrib/aws.rb:28:in `call'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-aws-0.2.0/lib/miasma/contrib/aws.rb:28:in `all_result_pages'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-aws-0.2.0/lib/miasma/contrib/aws/orchestration.rb:73:in `load_stack_data'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-aws-0.2.0/lib/miasma/contrib/aws/orchestration.rb:303:in `stack_all'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-0.2.38/lib/miasma/models/orchestration/stacks.rb:28:in `perform_population'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-0.2.38/lib/miasma/types/collection.rb:22:in `block in all'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-0.1.32/lib/bogo/memoization.rb:60:in `memoize'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/miasma-0.2.38/lib/miasma/types/collection.rb:21:in `all'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/provider.rb:169:in `block in fetch_stacks'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/cache.rb:265:in `block in locked_action'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/cache.rb:321:in `lock'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/cache.rb:264:in `locked_action'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/provider.rb:166:in `fetch_stacks'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/provider.rb:97:in `cached_stacks'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/provider.rb:92:in `stacks'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/command/list.rb:38:in `get_stacks'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-ui-0.1.12/lib/bogo-ui/table.rb:35:in `method_missing'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/command/list.rb:14:in `block (2 levels) in execute!'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-ui-0.1.12/lib/bogo-ui/table.rb:56:in `table'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/command/list.rb:13:in `block in execute!'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-ui-0.1.12/lib/bogo-ui/table.rb:78:in `instance_exec'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-ui-0.1.12/lib/bogo-ui/table.rb:78:in `display'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/lib/sfn/command/list.rb:31:in `execute!'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/bin/sfn:48:in `block (4 levels) in <top (required)>'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-cli-0.1.32/lib/bogo-cli/setup.rb:26:in `call'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-cli-0.1.32/lib/bogo-cli/setup.rb:26:in `block in bogo_cli_run'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/slop-3.6.0/lib/slop.rb:260:in `call'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/slop-3.6.0/lib/slop.rb:260:in `parse!'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/slop-3.6.0/lib/slop.rb:235:in `parse!'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/slop-3.6.0/lib/slop.rb:65:in `parse!'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/slop-3.6.0/lib/slop.rb:54:in `parse'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/bogo-cli-0.1.32/lib/bogo-cli/setup.rb:48:in `define'
/Users/doug/.chefdk/gem/ruby/2.1.0/gems/sfn-2.0.2/bin/sfn:14:in `<top (required)>'
/Users/doug/.chefdk/gem/ruby/2.1.0/bin/sfn:23:in `load'
/Users/doug/.chefdk/gem/ruby/2.1.0/bin/sfn:23:in `<main>'

[chrisroberts]

@dougireton hi! I don't believe you are experiencing the same problem. This issue relates to the instance_profile implementation for ec2 instance authentications. Looking at your configuration I believe you probably want to remove the token entries and just provide:

aws_sts_role_arn "ROLE_ARN_TO_ASSUME"

Miasma will make the requests to the STS API using your provided credentials to assume the provided role and generate tokens as required for the API interactions. If that change doesn't help, please let me know. Thanks!

[chrisroberts]

And for the instance_profile issue, that work is happening here: https://github.com/miasma-rb/miasma-aws/tree/fix/sts

[dougireton]

@chrisroberts Sorry but which credentials do I need to provide? Do you mean these?

aws_access_key_id=<MY_VALID_KEY>
aws_secret_access_key=<my_secret_access_key>

[chrisroberts]

Yeah, you'll need to provide 4 things:

• aws_access_key_id
• aws_secret_access_key
• aws_region
• aws_sts_role_arn

These can be provided from the the AWS configuration files in your home directory, from the .sfn configuration file directly, or a mix of both (the .sfn file will have precedence). The aws_sts_role_arn is the AWS ARN for the role you are to assume using your own credentials.

[oifland]

In my scenario, I have a service that already provides the access_key/secret/session token combination and the region is fixed. Is there a way to just use that token instead of trying to assume a role?

[chrisroberts]

Interesting. I'm not familiar with all the use cases around sts so this is helpful! Ideally then, if the sts token is provided directly, all checks should be automatically disabled. Right now a check is performed prior to request to validate that the current token is not past its expiry date, and will regenerate a new token if expired. I'll add in a check for explicitly provided tokens so it side steps those and get a release pushed out here shortly.

[chrisroberts]

Fix and enhancement added in release 0.2.2.

Thanks everyone!

[oifland]

This is great! I'll open a new issue for one other fix I had to do to get things working from the credentials file.