Aws::STS::Errors::InvalidClientTokenId

[korncola]

Describe the bug

when creating an ec2 client with ruby-sdk with a profile i access via an assumed role in a region that is not enabled by default (in my case eu-south-1) i get following error:

ec2 = Aws::EC2::Client.new(region: 'eu-south-1')
Aws::STS::Errors::InvalidClientTokenId: The security token included in the request is invalid.
 /home/korncola/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/aws-sdk-core-3.171.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'

• used profile is in~/.aws/config
• profile accessed via OrganizationalAccessRole
• non default enabled region eu-south-1
• works for default enabled regions like 'eu-west-1'
• same config & profile works in aws-cli with eu-south-1: e.g. aws ec2 describe-instances --region eu-south-1
• aws-sdk gem updated, using aws-sdk ~>3 in Gemfile, requiring only 'aws-sdk'

Expected Behavior

creating a client in non default regions works

Current Behavior

see above

Reproduction Steps

require 'aws-sdk' ec2 = Aws::EC2::Client.new(region: 'eu-south-1')

Possible Solution

No response

Additional Information/Context

No response

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk

Environment details (Version of Ruby, OS environment)

Ruby 3.1.2, Arch Linux

[mullermp]

Sorry you're running into this. I won't be able to reproduce this unless you can provide a snippet of your config file, but you'll need to remove any sensitive information. Are you using the default profile? The error you are getting back is a service error from STS and not an error from the Ruby SDK. Are you sure the correct profile is being loaded? You can use the profile option when creating a client. Please also pass http_wire_trace: true into your EC2 client and verify that STS assume role is being called with the credentials you expect (look at the Authorization header).

[korncola]

i think its cause non default enabled regions use different tokens (for some silly reason). The same config and profile work with aws cli, its just with the sdk. I am loading the profile with setting the environment variables, a call with aws cli works, with ruby sdk in pry not.

[korncola]

config:

[AUTH-ACCOUNT]
aws_account_id = 1337
mfa_serial = arn:aws:iam::1337:mfa/korncola
[profile PLAY1]
role_arn = arn:aws:iam::1338:role/OrganizationAccountAccessRole
color = c67116
region = eu-west-2
source_profile = AUTH-ACCOUNT
cli_pager = 

in my shell i switch profiles with following script, it basically setting for this example AWS_PROFILE=PLAY:

#!/bin/zsh
select p in $(grep -oP '(?<=\[profile).*?(?=\])' ~/.aws/config); do
     export AWS_PROFILE=$p
     zsh
done

in action: after switching profile my prompt shows me active profile: korncola@rakete  ~ AWS: PLAY1

proof:

 echo $AWS_PROFILE                                                                                                                                                                                                   
PLAY1

aws ec2 describe-instances --region 'eu-south-1' gives correct answer wtih some instances running in Milano

calling pry in same shell and requiring 'aws-sdk':

korncola@rakete  ~ AWS: PLAY1
 ❯ pry                                                                                                                                                                                                                 
3.1.2 (main):0 > require 'aws-sdk'
=> true
3.1.2 (main):0 > ec2 = Aws::EC2::Client.new(region: 'eu-south-1', http_wire_trace: true)
Aws::STS::Errors::InvalidClientTokenId: The security token included in the request is invalid.
from /home/korncola/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/aws-sdk-core-3.171.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
3.1.2 (main):0 > ec2 = Aws::EC2::Client.new(region: 'eu-west-1', http_wire_trace: true)
=> #<Aws::EC2::Client>

eu-west-1 -> works eu-south-1 -> error

cli with eu-south-1 -> works

same session in shell

[korncola]

i found it. EDIT: it was not the regional_endpoint setting, it was the managment account wich had not the region enabled, but the sub account had. Dont turn 2 screws at the same time... set in the config inside my AUTH/default profile: sts_regional_endpoints = regional

https://docs.aws.amazon.com/sdkref/latest/guide/feature-sts-regionalized-endpoints.html

After that can create clients in non default regions (that are enabled in the account of course). Yeah!

I think the issue is, as stated in the linked document: all sdk are defaulting now to regional sts which use different tokens, while my environment was using the default legacy mode and thus expecting different tokens, as they differ in size. Of course the document also says, for non default regions it will automatically use regional endpoints. The CLI works as expected even without this fix, the SDK for some reason not. So maybe there is still something wrong in the SDK regarding this. At least so far my issue is solved. @mullermp thnaks for your reply anyway!

[mullermp]

Thanks for responding back. I was able to reproduce the same failure behavior with CLI (my account does not have eu-south-1 enabled for STS). I think this is definitely intentional by STS. I don't know how you were able to get CLI to work in that same case. The error is returned by the service.

This fails:

aws sts get-caller-identity --region eu-south-1

and with Ruby, this fails too:

Aws::STS::Client.new(region: 'eu-south-1').get_caller_identity

[mullermp]

Closing as this is intentional by the service. Please re-open if there's still issue or concern with the SDK.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.