AWS Cross Account IAM Access Key Rotation with Commander using EC2 Instance Role

[apate007]

Wondering if this feature will be available in the future.

It appears that using commander running on EC2, the 'Login' field is used for finding the account in the iam::ListAccessKeys API call in keepercommander/plugins/awskey/aws_accesskey.py. While this works for same account API Calls, it does not work for cross account because the 'Distinguished Name' from the record containing the AWS arn is not used to find which account this record belongs to. I noticed however that cross account roles can be used for the in-GUI password rotation. I understand this could be a limitation caused by the IAM ListAccessKeys CLI command but it would be great to have the ability for cross account IAM Access Key rotation available while running Keeper Commander on an EC2 instance.

Associated error when using assume role to another account in coordination with the Commander rotate xxxxx --plugin awskey command:

>>> [RQ] vault/sync_down: {
  "continuationToken": ""
}
Starting new HTTPS connection (1): keepersecurity.com:443
https://keepersecurity.com:443 "POST /api/rest/vault/sync_down HTTP/11" 200 83
>>> [RS] vault/sync_down: {
  "continuationToken": ""
}
Decrypting meta data keys
Decrypting team keys
Decrypting shared folder keys
Resolve record keys. Meta data
Decrypting records
Decrypting non shared data
Decrypting folders
No rules, just generate
Rotating AWS access key id "xxx" for user "xxx"
Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
Changing event name from before-call.apigateway to before-call.api-gateway
Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
IMDS ENDPOINT: 
Looking for credentials via: env
Looking for credentials via: assume-role
Looking for credentials via: assume-role-with-web-identity
Looking for credentials via: sso
Looking for credentials via: shared-credentials-file
Looking for credentials via: custom-process
Looking for credentials via: config-file
Looking for credentials via: ec2-credentials-file
Looking for credentials via: boto-config
Looking for credentials via: container-role
Looking for credentials via: iam-role
Starting new HTTP connection (1): 
 "PUT /latest/api/token HTTP/11" 200 56
Resetting dropped connection: x.x.x.x
 "GET /latest/meta-data/iam/security-credentials/ HTTP/11" 200 22
Resetting dropped connection: x.x.x.x
 "GET /latest/meta-data/iam/security-credentials/<RotationRole> HTTP/11" 200 1586
Found credentials from IAM Role: <RotationRole>
Loading JSON file: C:\Program Files (x86)\Keeper Commander\_internal\botocore\data\endpoints.json
Loading JSON file: C:\Program Files (x86)\Keeper Commander\_internal\botocore\data\sdk-default-configuration.json
Event choose-service-name: calling handler <function handle_service_name_alias at 0x000001F9AEC61DA0>
Loading JSON file: C:\Program Files (x86)\Keeper Commander\_internal\botocore\data\iam\2010-05-08\service-2.json.gz
Loading JSON file: C:\Program Files (x86)\Keeper Commander\_internal\botocore\data\iam\2010-05-08\endpoint-rule-set-1.json.gz
Loading JSON file: C:\Program Files (x86)\Keeper Commander\_internal\botocore\data\partitions.json
Event creating-client-class.iam: calling handler <function add_generate_presigned_url at 0x000001F9AEBDB7E0>
Looking for endpoint for iam via: environment_service
Looking for endpoint for iam via: environment_global
Looking for endpoint for iam via: config_service
Looking for endpoint for iam via: config_global
No configured endpoint found.
Setting iam timeout as (60, 60)
Loading JSON file: C:\Program Files (x86)\Keeper Commander\_internal\botocore\data\_retry.json
Registering retry handlers for service: iam
Created AWS session using AWS profile "default"
Event before-parameter-build.iam.ListAccessKeys: calling handler <function generate_idempotent_uuid at 0x000001F9AECA07C0>
Calling endpoint provider with parameters: {'Region': 'aws-global', 'UseDualStack': False, 'UseFIPS': False}
Endpoint provider result: https://iam.amazonaws.com
Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 'iam'}
Event before-call.iam.ListAccessKeys: calling handler <function add_recursion_detection_header at 0x000001F9AEC7AF20>
Event before-call.iam.ListAccessKeys: calling handler <function inject_api_version_header_if_needed at 0x000001F9AECA22A0>
Making request for OperationModel(name=ListAccessKeys) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.35.26 md/Botocore#1.35.26 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.11.9 md/pyimpl#CPython exec-env/EC2 cfg/retry-mode#legacy Botocore/1.35.26'}, 'body': {'Action': 'ListAccessKeys', 'Version': '2010-05-08', 'UserName': 'xxxx'}, 'url': 'https://iam.amazonaws.com/', 'context': {'client_region': 'aws-global', 'client_config': <botocore.config.Config object at 0x000001F9AFC7FA90>, 'has_streaming_input': False, 'auth_type': 'v4', 'unsigned_payload': None, 'signing': {'region': 'us-east-1', 'signing_name': 'iam'}, 'endpoint_properties': {'authSchemes': [{'name': 'sigv4', 'signingName': 'iam', 'signingRegion': 'us-east-1'}]}}}
Event request-created.iam.ListAccessKeys: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000001F9AFC7FA50>>
Event choose-signer.iam.ListAccessKeys: calling handler <function set_operation_specific_signer at 0x000001F9AECA05E0>
Calculating signature using v4 auth.
CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:iam.amazonaws.com
x-amz-date:20241008T132539Z
x-amz-security-token:IQoJb3JpZ2luX2VjEO3//////////
content-type;host;x-amz-date;x-amz-security-token
ffef911ef310d220d706bb1
StringToSign:
AWS4-HMAC-SHA256
20241008T132539Z
20241008/us-east-1/iam/aws4_request

Signature:

Event request-created.iam.ListAccessKeys: calling handler <function add_retry_headers at 0x000001F9AECA2AC0>
Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://iam.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'Boto3/1.35.26 md/Botocore#1.35.26 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.11.9 md/pyimpl#CPython exec-env/EC2 cfg/retry-mode#legacy Botocore/1.35.26', 'X-Amz-Date': b'20241008T132539Z', 'X-Amz-Security-Token': b'IQoJb3JpZ2luX2VjEO3//////////Uf7FqSHThylENbRKTnzlVt+e/jr+VyRvUsK+J6DykV1wqyOH/k2qsWQIywCuXpOxIQc5cOgi4xF4WYFQHHkn7EOoHq/jlRGYHorgFvTncZ432pXjZCyLxnJ4f5M55+rC7jj+ZM+VjybuDoRkZkvxkH+Fg8/6h1h9z7LdngJN5BVxQmE7n4K8wr94M8m0EyeNj0DQg2+VLAMCbqd31At2FNhPTKw4B4wnQv/5ENfrV4fuWrmTKEj9dN2qQV6s2+tSI9tFJsuJdfLLf9xHwpA4qmGtI14ZW4P5i38C8j6n/z9IGKpeTn0DMSty8XCl1OsgFwvLBmk1wPu12acq9dbeS7A3BMoiidH0tx1bnK5pc2GSJuzkDyKP/AAwCLdwMZjZIUSr8hge4BZGVlZqOY4bF9TTzVm0D26NrFIF6JaD0UusqZbX+gDlnQXiFJiCIrrbNQw919RtH0xHqHpNkR96ufc00RsWRNkjXfKD92pdU8aIW8Xn+G08nvbKteaM9/IFZi8iiJQKdb0XVrj3iszHHw65TUbDDvzJS4BjqxAcVKMr399IJ0k64CEu5uagWyOtDsJ3cW4AwuPNTcmNAaxbC9luipggZAbzI1+hu3TsKodHhriBTGEwOd/A6ys9mfZINZJf2oJUB1l5+XToKq/JPTrMlqMi13C6++JpFHz2i7yfD4bZ9R+WOJb8yzCNAHs5l5cPfOuO/zSg+jZzQEk3+sodnhDG2PdRJRImRe/MczAlDLut72CFAmqZthrDjVrxxXSRQlmzNWTY1lEztXVw==', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAVKF3Q2ZLR3LHXIUQ/20241008/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=935745b61c71141f969edda2a44eaf2cd4851d31eae3316d2eec7ee3f101f275', 'amz-sdk-invocation-id': b'f5d440e9-5d72-49be-a6b1-5c6210f60b01', 'amz-sdk-request': b'attempt=1', 'Content-Length': '60'}>
Certificate path: C:\Program Files (x86)\Keeper Commander\_internal\certifi\cacert.pem
Starting new HTTPS connection (1): iam.amazonaws.com:443
https://iam.amazonaws.com:443 "POST / HTTP/11" 404 290
Response headers: {'Date': 'Tue, 08 Oct 2024 13:25:40 GMT', 'x-amzn-RequestId': '8cfb820a-2a8d-48a0-9d00-ef0107a1f759', 'Content-Type': 'text/xml', 'Content-Length': '290'}
Response body:
b'<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>NoSuchEntity</Code>\n    <Message>The user with name xxxx cannot be found.</Message>\n  </Error>\n  <RequestId>8cfb820a-2a8d-48a0-9d00-ef0107a1f759</RequestId>\n</ErrorResponse>\n'
Response headers: {'Date': 'Tue, 08 Oct 2024 13:25:40 GMT', 'x-amzn-RequestId': '8cfb820a-2a8d-48a0-9d00-ef0107a1f759', 'Content-Type': 'text/xml', 'Content-Length': '290'}
Response body:
b'<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>NoSuchEntity</Code>\n    <Message>The user with name xxxx cannot be found.</Message>\n  </Error>\n  <RequestId>8cfb820a-2a8d-48a0-9d00-ef0107a1f759</RequestId>\n</ErrorResponse>\n'
Event needs-retry.iam.ListAccessKeys: calling handler <botocore.retryhandler.RetryHandler object at 0x000001F9AFC94990>
No retry needed.
Event after-call.iam.ListAccessKeys: calling handler <function json_decode_policies at 0x000001F9AECA1440>
Error listing existing AWS keys for user "xxxx": An error occurred (NoSuchEntity) when calling the ListAccessKeys operation: The user with name xxxx cannot be found.
Password rotation failed for record "IAM Account: xxxx" (uid=[<RecordUID>]), plugin "awskey".
My Vault>

[Keeper-Eric]

This can be done using Keeper Secrets Manager Password Rotation with a PAM script attached.

[apate007]

This can be done using Keeper Secrets Manager Password Rotation with a PAM script attached.

Hi @Keeper-Eric, thank you for the response. This method does not work for our use case. Wrote this issue in hopes of requesting this as a feature for Keeper Commander since cross-account AWS Credential rotation is present for passwords only as part of the Keeper Secrets Manager Gui, but not for passwords or awskey rotation in Commander.

[sk-keeper]

The Commander team will fix this issue in the next release.

[apate007]

The Commander team will fix this issue in the next release.

Thanks @sk-keeper!

[sk-keeper]

The changes are in the release branch. awskey password rotation plugin accepts a new parameter cmdr:aws_assume_role If set then the plugin assumes this role prior to calling any iam access key method.

The Commander is going to be released this Friday.

[sk-keeper]

Commander 16.11.15 has been released.

[apate007]

Commander 16.11.15 has been released.

This works as expected. Thanks for adding this so quickly @sk-keeper!