"The security token included in the request is invalid"

[LinguineCode]

I couldn't find a related issue on this. I am getting this error:

The error

An error occurred: IamRoleLambdaExecution - The security token included in the request is invalid (Service: AmazonIdentityManagement; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: a42d8546-3142-11e9-8b89-ef3966ba0855).

My use case

I am running aws-vault exec myprofile -- longrunningscript.sh. It could take longrunningscript.sh about 3-5 minutes before it gets to the point where it hits the AWS API.

My workaround

I have been doing AWS_PROFILE=myprofile longrunningscript.sh just to get by, but this isn't going to work all the time

Additionally: A README question

According to the README.md it says:

These expire in a short period of time, so the risk of leaking credentials is reduced.

What exactly is a short period of time? Is this my issue? Is there any way to extend it?

[osterman]

We also started getting this error. Slightly different invocation.

aws-vault exec xxxx-xxx-admin --server  -- aws s3 ls
Enter passphrase to unlock /conf/.awsvault/keys/:

An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.

While the same exact command works if we remove the --server arg. This stopped working at some point in the last couple of weeks, but cannot pin point it.

[geertn]

I'm seeing this too when running terraform using aws-vault on two separate machines (of colleagues), both OSX. On my Linux machine I haven't seen this error.

Tried regenerating the Access Keys, restart shell sessions, nothing seems to help.

[lox]

I'd guess all of these things are totally different errors and causes. That is pretty much AWS' catch all error.

@geertn to narrow down the problem try running aws-vault exec <profile> -- aws configure and see what you get.

[geertn]

prompt$ aws-vault exec eks-pilot -- aws configure
AWS Access Key ID [****************]: xxxxxxxxx
AWS Secret Access Key [****************]:xxxxxxxxxxx
Default region name [ap-northeast-1]: eu-west-1
Default output format [json]: 

When my colleague/customer sets the aws api credentials hard in environment it does work.

$ export AWS_ACCESS_KEY_ID="anaccesskey"
$ export AWS_SECRET_ACCESS_KEY="asecretkey"
$ export AWS_DEFAULT_REGION="eu-west-1"

[lox]

Argh, apologies, I meant aws configure list

You'll now want to check you don't have anything in your ~/.aws/credentials file as that will trump anything that aws-vault puts out.

[geertn]

Thanks for taking the time to help with this, even though it might just as well be not related to aws-vault.

$ aws-vault exec eks-pilot -- aws configure list

      Name                    Value             Type    Location

      ----                    -----             ----    --------

   profile                <not set>             None    None

access_key     ****************YDEK              env    

secret_key     ****************xR8H              env    

    region                eu-west-1              env    AWS_DEFAULT_REGION

$ aws configure list

      Name                    Value             Type    Location

      ----                    -----             ----    --------

   profile                <not set>             None    None

access_key     ****************YBZF              env    

secret_key     ****************DzbN              env    

    region                eu-west-1              env    AWS_DEFAULT_REGION

Removing ~/.aws/credentials en ~/.aws/config didn't help

[geertn]

I thought I'd try it with the new aws-vault release.

(awscli) geertn@asdfdskj:~$ aws-vault  --version
v4.6.0
(awscli) geertn@asdfdskj:~$ 

First run of

aws-vault exec --debug mm-assume_gn -- aws ec2 describe-instances

Resulted in the error again. Second run did work as expected strange enough. I do see the error consistently when doing a rotate:

(awscli) geertn@asdfdskj:~$ aws-vault rotate --debug mm-assume_gn -- 
2019/06/21 13:23:23 [keyring] Considering backends: [kwallet secret-service pass file]
2019/06/21 13:23:23 [keyring] Failed backend kwallet: The name org.kde.kwalletd was not provided by any .service files
2019/06/21 13:23:23 Loading config file /home/geertn/.aws/config
2019/06/21 13:23:23 Parsing config file /home/geertn/.aws/config
Rotating credentials for profile "mm-assume_gn" (takes 10-20 seconds)
2019/06/21 13:23:23 Looking up keyring for mm-assume_gn
2019/06/21 13:23:23 Found old access key  ****************JJKG for user MM_Assume_GeertNijpels
2019/06/21 13:23:23 Skipping session token and using master credentials directly
2019/06/21 13:23:23 Using old credentials to create a new access key
2019/06/21 13:23:24 Created new access key
2019/06/21 13:23:24 Using new credentials to delete the old new access key
2019/06/21 13:23:24 Waiting for new IAM credentials to propagate (takes up to 10 seconds)
2019/06/21 13:23:24 Skipping session token and using master credentials directly
2019/06/21 13:23:29 Retrying after error: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: fbaab1dc-9416-11e9-bb7b-c1e9cddbe175
2019/06/21 13:23:29 Skipping session token and using master credentials directly
2019/06/21 13:23:35 Retrying after error: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: fed90f09-9416-11e9-bb7b-c1e9cddbe175
2019/06/21 13:23:35 Skipping session token and using master credentials directly
2019/06/21 13:23:35 Looking for sessions for mm-assume_gn
2019/06/21 13:23:35 Looking up all keys in keyring
2019/06/21 13:23:35 Rotated credentials for profile "mm-assume_gn" in vault
Done!
(awscli) geertn@asdfdskj:~$ 

[mr337]

I am having the same problem but what looks to be IAM related actions under IAM. Oddly enough the rotate action works fine. Everything else works (ec2, s3, dynamodb, cf) just to test. Currently using version 4.6.0.

When exporting env keys everything works.

[mtibben]

@solsglasses Session TTLs are documented at https://github.com/99designs/aws-vault/blob/master/USAGE.md#assuming-a-role-for-more-than-1h

Feel free to update the docs in a PR if you think it could be clearer

[mamash]

I am having the same problem but what looks to be IAM related actions under IAM. Oddly enough the rotate action works fine. Everything else works (ec2, s3, dynamodb, cf) just to test. Currently using version 4.6.0.

When exporting env keys everything works.

I, too, am facing this problem when wrapping around Ansible calls. All modules work fine, but those that interact with IAM are failing, as if the ephemeral credentials weren't enough.

[plektra]

I'm using aws-vault version 5.3.2 and having similar issues with Serverless Framework. When executing "sls deploy", the corresponding CloudFormation update operation fails when it tries to update an IAM role, due to following error:

The security token included in the request is invalid (Service: AmazonIdentityManagement; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: asdfafsddfasdasfasfdafsd)

To work around this issue I'm using --no-session for now.

[frezbo]

@plektra if you need access to IAM operations with a session token, it should be MFA authenticated. It's a requirement from AWS. Please add mfa_serial propery in the aws config file, then aws-vault will prompt for MFA token,

[plektra]

@frezbo Oh interesting, thanks for the tip. I didn't know AWS requires MFA-authenticated session in order to do IAM actions. Now it works a-ok, cool. :)

[mtibben]

Thanks @frezbo @plektra, if you can update USAGE.md to make this common gotcha easier to find that would be excellent

[joshuablanco]

In linux

1. make sure your credentials file is all set.
2. make sure aws configure list is working and brings consequent information.
3. If you are using sam export the variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGIO, with super user clearance.
4. The execution with the aws toolkit it seems to check those environment variables according the credentials file.

[jpcaparas]

Thanks, setting the mfa_serial was handy in troubleshooting the cdk deploy error I was getting:

1:48:19 pm | CREATE_FAILED        | AWS::IAM::Role        | HelloHandlerServiceRole11EFXXXX
The security token included in the request is invalid (Service: AmazonIdentityManagement; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: <redacted>
; Proxy: null)

 ❌  CdkWorkshopStack failed: Error: The stack named CdkWorkshopStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
    at Object.waitForStackDeploy (/opt/homebrew/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:305:11)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at prepareAndExecuteChangeSet (/opt/homebrew/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:352:26)
    at CdkToolkit.deploy (/opt/homebrew/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:189:24)
    at initCommandLine (/opt/homebrew/lib/node_modules/aws-cdk/bin/cdk.ts:225:9)
The stack named CdkWorkshopStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE

[TheAshwanik]

In linux

1. make sure your credentials file is all set.
2. make sure aws configure list is working and brings consequent information.
3. If you are using sam export the variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGIO, with super user clearance.
4. The execution with the aws toolkit it seems to check those environment variables according the credentials file.

Just wondering. If I export the access_key and secret_key , then whats the point of using aws_vault? I thought aws_vault was supposed to help us not exporting or hardcoding the secrets in credentials file?

Trying to understand. May be I am missing something?