Closed FernandoMiguel closed 5 years ago
Any luck on this? I'm running into the exact same problem. I've used this successfully in the past, so I wonder if there was a change in how AWS handles the empty requestParameters?
What version are you on @clintoncampbell? This isn't a bug with aws-vault, it's generally a case of your user not having the correct permissions.
Both @clintoncampbell and @FernandoMiguel, could you explain how your credential setup looks? Are you assuming roles? Did you use the root credentials, or an iam user or an assumed role to try and rotate permissions? Did you verify that your user DOES have permission for iam:CreateAccessKey
?
I'm using 4.1.0. Absolutely, users are attached to a Humans group with an inline policy giving them the ability to manage passwords and access keys directly:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:*LoginProfile",
"iam:*AccessKey*",
"iam:*SSHPublicKey*"
],
"Resource": "arn:aws:iam::ACCOUNTID:user/${aws:username}"
},
{
"Effect": "Allow",
"Action": [
"iam:ListAccount*",
"iam:GetAccountSummary",
"iam:GetAccountPasswordPolicy",
"iam:ListUsers"
],
"Resource": "*"
}
]
}
Replicating the process through the AWS CLI without aws-vault works as long as I specify a user, e.g., aws iam list-access-keys --profile just_keys --user ccampbell
. If I omit the --user flag, I've been seeing the same error message I was seeing when using aws-vault rotate
.
All that said, I think it may have been an incorrect account id in the policy on my end.
Yep, I can confirm that rotate is working correctly since I fixed the account ID in this policy.
$ aws-vault --version v4.1.0
$ aws-vault --debug rotate iam
2017/12/05 09:25:15 Loading config file /Users/fernando/.aws/config
2017/12/05 09:25:15 Parsing config file /Users/fernando/.aws/config
Rotating credentials for profile "iam"
2017/12/05 09:25:15 Looking up keyring for iam
2017/12/05 09:25:18 Found old access key ****************RFFQ for user fernando
2017/12/05 09:25:18 Skipping session token and using master credentials directly
2017/12/05 09:25:18 Using old credentials to create a new access key
aws-vault: error: Failed to get credentials for iam: AccessDenied: User: arn:aws:iam::IAMACCOUNTID:user/fernando is not authorized to perform: iam:CreateAccessKey on resource: user nullfernando with an explicit deny
no idea what nullfernando
is ...
this process never even asked for MFA token
this is our IAM account, so i am not assuming a role just yet. every other profile is a role.
this is our MFA policy, which is as referred in the OP an exact copy of aws provided policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAllUsersToListAccounts",
"Effect": "Allow",
"Action": [
"iam:ListAccountAliases",
"iam:ListUsers",
"iam:GetAccountPasswordPolicy",
"iam:GetAccountSummary"
],
"Resource": "*"
},
{
"Sid": "AllowIndividualUserToSeeAndManageOnlyTheirOwnAccountInformation",
"Effect": "Allow",
"Action": [
"iam:ChangePassword",
"iam:CreateAccessKey",
"iam:CreateLoginProfile",
"iam:DeleteAccessKey",
"iam:DeleteLoginProfile",
"iam:GetLoginProfile",
"iam:ListAccessKeys",
"iam:UpdateAccessKey",
"iam:UpdateLoginProfile",
"iam:ListSigningCertificates",
"iam:DeleteSigningCertificate",
"iam:UpdateSigningCertificate",
"iam:UploadSigningCertificate",
"iam:ListSSHPublicKeys",
"iam:GetSSHPublicKey",
"iam:DeleteSSHPublicKey",
"iam:UpdateSSHPublicKey",
"iam:UploadSSHPublicKey"
],
"Resource": "arn:aws:iam::IAMACCOUNTID:user/${aws:username}"
},
{
"Sid": "AllowIndividualUserToListOnlyTheirOwnMFA",
"Effect": "Allow",
"Action": [
"iam:ListVirtualMFADevices",
"iam:ListMFADevices"
],
"Resource": [
"arn:aws:iam::IAMACCOUNTID:mfa/*",
"arn:aws:iam::IAMACCOUNTID:user/${aws:username}"
]
},
{
"Sid": "AllowIndividualUserToManageTheirOwnMFA",
"Effect": "Allow",
"Action": [
"iam:CreateVirtualMFADevice",
"iam:DeleteVirtualMFADevice",
"iam:EnableMFADevice",
"iam:ResyncMFADevice"
],
"Resource": [
"arn:aws:iam::IAMACCOUNTID:mfa/${aws:username}",
"arn:aws:iam::IAMACCOUNTID:user/${aws:username}"
]
},
{
"Sid": "AllowIndividualUserToDeactivateOnlyTheirOwnMFAOnlyWhenUsingMFA",
"Effect": "Allow",
"Action": [
"iam:DeactivateMFADevice"
],
"Resource": [
"arn:aws:iam::IAMACCOUNTID:mfa/${aws:username}",
"arn:aws:iam::IAMACCOUNTID:user/${aws:username}"
],
"Condition": {
"Bool": {
"aws:MultiFactorAuthPresent": "true"
}
}
},
{
"Sid": "BlockMostAccessUnlessSignedInWithMFA",
"Effect": "Deny",
"NotAction": [
"iam:CreateVirtualMFADevice",
"iam:DeleteVirtualMFADevice",
"iam:ListVirtualMFADevices",
"iam:EnableMFADevice",
"iam:ResyncMFADevice",
"iam:ListAccountAliases",
"iam:ListUsers",
"iam:ListSSHPublicKeys",
"iam:ListAccessKeys",
"iam:ListServiceSpecificCredentials",
"iam:ListMFADevices",
"iam:GetAccountSummary",
"sts:GetSessionToken",
"iam:ChangePassword"
],
"Resource": "*",
"Condition": {
"BoolIfExists": {
"aws:MultiFactorAuthPresent": "false"
}
}
}
]
}
iam:UpdateAccessKey and iam:CreateAccessKey are in there.
I wonder if the fact that i never got to enter the MFA is the reason for the failure?
from cloud trail
{
"eventVersion": "1.02",
"userIdentity": {
"type": "IAMUser",
"principalId": "XXXXX",
"arn": "arn:aws:iam::IAMACCOUNTID:user/fernando",
"accountId": "IAMACCOUNTID",
"accessKeyId": "****************RFFQ",
"userName": "fernando"
},
"eventTime": "2017-12-05T09:25:18Z",
"eventSource": "iam.amazonaws.com",
"eventName": "CreateAccessKey",
"awsRegion": "us-east-1",
"sourceIPAddress": "XXXX",
"userAgent": "aws-sdk-go/1.4.14 (go1.9.1; darwin; amd64)",
"errorCode": "AccessDenied",
"errorMessage": "User: arn:aws:iam::IAMACCOUNTID:user/fernando is not authorized to perform: iam:CreateAccessKey on resource: user nullfernando with an explicit deny",
"requestParameters": null,
"responseElements": null,
"eventType": "AwsApiCall",
"recipientAccountId": "IAMACCOUNTID"
}
this nullfernando
seems to be the reason for it failing
was looking at https://github.com/99designs/aws-vault/pull/180/files and i'm missing iam:GetUser could that be it ?
no luck with that
Have you tried replicating the process from the CLI to ensure the individual steps are working?
@FernandoMiguel This could also have to do with the manner in which you've set up your profiles in config. I'm curious whether your iam
profile has a role attached or just creds? Based on https://github.com/99designs/aws-vault/issues/182, you would need to rotate using the role profile rather than a creds profile for a user with MFA attached.
this is my ~/.aws/config @clintoncampbell
[profile ORG-iam]
region=eu-west-1
mfa_serial=arn:aws:iam::ACCOUNTID:mfa/fernando
[profile ORG-iam-admin]
region=eu-west-1
role_arn=arn:aws:iam::ACCOUNTID:role/IAM-Admin-role
source_profile=ORG-iam
mfa_serial=arn:aws:iam::ACCOUNTID:mfa/fernando
[profile ORG-readonly]
region=eu-west-1
role_arn=arn:aws:iam::ACCOUNTID02:role/ORG-IAM-Readonly
source_profile=ORG-iam
mfa_serial=arn:aws:iam::ACCOUNTID:mfa/fernando
[profile ORG-tools]
region=eu-west-1
role_arn=arn:aws:iam::ACCOUNTID03:role/OrganizationAccountAccessRole
source_profile=ORG-iam
mfa_serial=arn:aws:iam::ACCOUNTID:mfa/fernando
[profile ORG-poc]
region=eu-west-1
role_arn=arn:aws:iam::ACCOUNTID04:role/OrganizationAccountAccessRole
source_profile=ORG-iam
mfa_serial=arn:aws:iam::ACCOUNTID:mfa/fernando
$ aws-vault list
Profile Credentials Sessions
======= =========== ========
ORG-iam ORG-iam -
ORG-iam-admin ORG-iam -
ORG-readonly ORG-iam -
ORG-tools ORG-iam -
ORG-poc ORG-iam -
so trying to change the key for the 1st profile which is the one with the keys
resource: user nullfernando
This is an AWS bug as far as I could tell.
Could I trouble you to try this with the latest master @FernandoMiguel?
I'll try to give it a go tomorrow
-- Fernando Miguel
@lox what's the easiest way to grant access to the docker container to the keychain ?
As in, how do you access your macOS keychain from a linux docker container? If so, the answer would be about 3 years of development :)
Ahaha I'll make a new key then and add it to Docker
But first, it needs to work, and I've opened another ticket for some issues the Dockerfile is having
@lox can you take a look at #209 ? it's preventing me from testing master thanks
Ping @lox
Let me know if you get a chance to test master
! Happy to help debug.
I did try via compiling from source in Docker but was broken. I opened another issue for it
Currently away from my laptop with no chance to test it But did have a few devs following a guide I wrote and some tried to rotate keys and it failed as usual
This was with version 4.1 just a couple of days before 4.2 come out
Is your laptop macOS? I can walk you through getting enough golang on there to run it if that helps.
I can put go there. Usually avoid pestering extra build tools into the host.
I'll give it a go
$ aws-vault --debug --no-session rotate iam
2018/01/23 10:07:08 [keyring] Considering backends: [keychain file]
2018/01/23 10:07:08 Loading config file /Users/fernando/.aws/config
2018/01/23 10:07:08 Parsing config file /Users/fernando/.aws/config
aws-vault: error: unknown long flag '--no-session', try --help
$ aws-vault --debug rotate iam
2018/01/23 10:07:32 [keyring] Considering backends: [keychain file]
2018/01/23 10:07:32 Loading config file /Users/fernando/.aws/config
2018/01/23 10:07:32 Parsing config file /Users/fernando/.aws/config
Rotating credentials for profile "iam" (takes 10-20 seconds)
2018/01/23 10:07:32 Looking up keyring for iam
2018/01/23 10:07:32 [keyring] Querying keychain for service="aws-vault", account="iam", keychain="aws-vault.keychain"
2018/01/23 10:07:35 [keyring] Found item "aws-vault (iam)"
2018/01/23 10:07:35 Found old access key **************** for user fernando
2018/01/23 10:07:35 Skipping session token and using master credentials directly
2018/01/23 10:07:35 Using old credentials to create a new access key
aws-vault: error: Failed to get credentials for iam: AccessDenied: User: arn:aws:iam::1234567890:user/fernando is not authorized to perform: iam:CreateAccessKey on resource: user fernando with an explicit deny
status code: 403, request id: 3c1048c4-0025-11e8-9ab8-c7490ee0d2d6
$ aws-vault --version
v4.2.0
testing 4.2 with --no-session to see if i could use @0xdabbad00 trick. no luck
but at least now the error is no longer nullfernando
let me try from master. i'll see if i try it with an user that doesnt have any restrictive MFA policies
$ aws-vault --debug rotate test
2018/01/23 10:15:17 [keyring] Considering backends: [keychain file]
2018/01/23 10:15:17 Loading config file /Users/fernando/.aws/config
2018/01/23 10:15:17 Parsing config file /Users/fernando/.aws/config
Rotating credentials for profile "test" (takes 10-20 seconds)
2018/01/23 10:15:17 Looking up keyring for test
2018/01/23 10:15:17 [keyring] Querying keychain for service="aws-vault", account="test", keychain="aws-vault.keychain"
2018/01/23 10:15:17 [keyring] Found item "aws-vault (test)"
2018/01/23 10:15:17 Found old access key ****************PRRA for user test
2018/01/23 10:15:17 Skipping session token and using master credentials directly
2018/01/23 10:15:17 Using old credentials to create a new access key
2018/01/23 10:15:18 Created new access key
2018/01/23 10:15:18 [keyring] Checking keychain status
2018/01/23 10:15:18 [keyring] Keychain status returned nil, keychain exists
2018/01/23 10:15:18 [keyring] Keychain item doesn't trust aws-vault
2018/01/23 10:15:18 [keyring] Adding service="aws-vault", label="aws-vault (test)", account="test", trusted=false to osx keychain "aws-vault.keychain"
2018/01/23 10:15:18 [keyring] Item already exists, deleting
2018/01/23 10:15:18 [keyring] Adding item again
2018/01/23 10:15:18 Using new credentials to delete the old new access key
2018/01/23 10:15:18 Waiting for new IAM credentials to propagate (takes up to 10 seconds)
2018/01/23 10:15:18 Skipping session token and using master credentials directly
2018/01/23 10:15:23 Retrying after error: InvalidClientTokenId: The security token included in the request is invalid.
status code: 403, request id: 4fca8693-0026-11e8-8667-d3eca4ef75b7
2018/01/23 10:15:23 Skipping session token and using master credentials directly
2018/01/23 10:15:28 Retrying after error: InvalidClientTokenId: The security token included in the request is invalid.
status code: 403, request id: 52d3cf05-0026-11e8-8667-d3eca4ef75b7
2018/01/23 10:15:28 Skipping session token and using master credentials directly
2018/01/23 10:15:28 Looking for sessions for test
2018/01/23 10:15:28 Looking up all keys in keyring
2018/01/23 10:15:28 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2018/01/23 10:15:28 [keyring] Found 4 results
2018/01/23 10:15:28 Session "test session (1234567890)" expires in 3h59m28.521648s
2018/01/23 10:15:28 Session "test session (1234567890)" matches profile "test"
2018/01/23 10:15:28 Removing keychain item service="aws-vault", account="test session (1234567890)", keychain "aws-vault.keychain"
2018/01/23 10:15:28 Deleted 1 existing sessions.
2018/01/23 10:15:28 Rotated credentials for profile "test" in vault
Done!
test account with no policies and no mfa, works
just added MFA to that user, and to my surprise, it never asked for it :O
i just applied this policy to user test https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_users-self-manage-mfa-and-creds.html
and still fails with
aws-vault: error: Failed to get credentials for test: AccessDenied: User: arn:aws:iam::1234567890:user/test is not authorized to perform: iam:CreateAccessKey on resource: user test with an explicit deny
Shouldn't you be using aws-vault --debug rotate iam-admin
?
I get the same error when I try and rotate my iam
profile.
@lox test user add Administrator profile attached
@lox what do you mean with iam profile? and by getting the same error, is this expected? how would i rotate my key?
just added MFA to that user, and to my surprise, it never asked for it :O
When you say "added MFA", what did you do?
Added a MFA key to the test user profile and to .aws/config
-- Fernando Miguel
On 23 Jan 2018 21:44, "Lachlan Donald" notifications@github.com wrote:
just added MFA to that user, and to my surprise, it never asked for it :O
When you say "added MFA", what did you do?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/99designs/aws-vault/issues/190#issuecomment-359941070, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKRrgPfXS5MG0zIA8M8rC9GnanwO9yrks5tNlK4gaJpZM4QkZMj .
🤔
The fact then that it didn't ask you for an MFA is perplexing. Perhaps that is the issue. Let me investigate.
Confirm the same issue aws-vault exec
asks for mfa but aws-vault login
does not. Running v4.1.0 on macOS.
Humm never tried login eheh All these recent tests were with v4.2
I still haven't had the chance to try master code. Maybe I can give it a go today... Maybe
@lox so i build from master source today. two new errors.
$ docker run --rm -it --entrypoint ash -v $PWD:/root/.aws/ aws-vault
/ # /aws-vault list
Profile Credentials Sessions
======= =========== ========
iam - -
/ # /aws-vault --debug add iam
2018/01/28 11:49:04 [keyring] Considering backends: [file]
2018/01/28 11:49:04 Loading config file /root/.aws/config
2018/01/28 11:49:04 Parsing config file /root/.aws/config
Enter Access Key ID: XXX
Enter Secret Access Key: XXX
2018/01/28 11:49:09 [keyring] Expanded file dir to /root/.awsvault/keys/
2018/01/28 11:49:09 [keyring] Expanded file dir to /root/.awsvault/keys/
Enter passphrase to unlock /root/.awsvault/keys/:
Added credentials to profile "iam" in vault
2018/01/28 11:49:19 Looking for sessions for iam
2018/01/28 11:49:19 Looking up all keys in keyring
2018/01/28 11:49:19 [keyring] Expanded file dir to /root/.awsvault/keys/
/ # /aws-vault list
Profile Credentials Sessions
======= =========== ========
iam iam -
/ # /aws-vault --debug exec iam --
2018/01/28 11:52:09 [keyring] Considering backends: [file]
2018/01/28 11:52:09 Loading config file /root/.aws/config
2018/01/28 11:52:09 Parsing config file /root/.aws/config
2018/01/28 11:52:09 Looking for sessions for iam
2018/01/28 11:52:09 Looking up all keys in keyring
2018/01/28 11:52:09 [keyring] Expanded file dir to /root/.awsvault/keys/
2018/01/28 11:52:09 Session not found in keyring for iam
2018/01/28 11:52:09 Looking up keyring for iam
2018/01/28 11:52:09 [keyring] Expanded file dir to /root/.awsvault/keys/
2018/01/28 11:52:09 [keyring] Expanded file dir to /root/.awsvault/keys/
Enter passphrase to unlock /root/.awsvault/keys/:
Enter token for arn:aws:iam::1234567890:mfa/fernando: XXX
2018/01/28 11:52:15 Using region "eu-west-1" from profile
2018/01/28 11:52:15 Getting new session token for profile iam
2018/01/28 11:52:15 Writing session for iam to keyring: "iam session (XXX35)"
2018/01/28 11:52:15 [keyring] Expanded file dir to /root/.awsvault/keys/
2018/01/28 11:52:15 [keyring] Expanded file dir to /root/.awsvault/keys/
2018/01/28 11:52:15 Using session ****************, expires in 3h59m57.072872405s
2018/01/28 11:52:15 Setting subprocess env: AWS_DEFAULT_REGION=eu-west-1, AWS_REGION=eu-west-1
2018/01/28 11:52:15 Setting subprocess env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
2018/01/28 11:52:15 Setting subprocess env: AWS_SESSION_TOKEN, AWS_SECURITY_TOKEN
aws-vault: error: fork/exec : no such file or directory
/ # /aws-vault --debug login iam
2018/01/28 11:52:32 [keyring] Considering backends: [file]
2018/01/28 11:52:32 Loading config file /root/.aws/config
2018/01/28 11:52:32 Parsing config file /root/.aws/config
2018/01/28 11:52:32 Skipping session token and using master credentials directly
2018/01/28 11:52:32 Looking up keyring for iam
2018/01/28 11:52:32 [keyring] Expanded file dir to /root/.awsvault/keys/
2018/01/28 11:52:32 [keyring] Expanded file dir to /root/.awsvault/keys/
Enter passphrase to unlock /root/.awsvault/keys/:
2018/01/28 11:52:34 No session token found, calling GetFederationToken
aws-vault: error: Failed to call GetFederationToken: AccessDenied: User: arn:aws:iam::1234567890:user/fernando is not authorized to perform: sts:GetFederationToken on resource: arn:aws:sts::1234567890:federated-user/fernando with an explicit deny
status code: 403, request id: b992afc8-0421-11e8-9330-95d74932ad24
Login for non-assumed roles depends on permission to call sts:GetFederationToken
i added this IAM inline policy but didn't change anything
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "sts-wildcard",
"Effect": "Allow",
"Action": "sts:*",
"Resource": "*"
}
]
}
I'm seeing the same thing here, tried on master
but still the same thing, is there anything I can do to help debug?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This was still happening last time I tried it with a restricted policy
Did you eventually figure it out @FernandoMiguel ? I'm having the same issue of on resource: user null<user_name>
(for a completely different system though), and this thread is the only thing that shows up with some google searches 😢 .
Did you eventually figure it out @FernandoMiguel ? I'm having the same issue of
on resource: user null<user_name>
(for a completely different system though), and this thread is the only thing that shows up with some google searches 😢 .
I don't recall @timotheeg . Haven't had issues in a long time
I've got the default MFA policy in place http://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_users-self-manage-mfa-and-creds.html
where's the cloudtrail via webconsole
and here a cloudtrail fail from aws-vault