Closed amitdwivedi-broadcom closed 5 months ago
Hi @amitdwivedi-broadcom - as mentioned here, these two issues you opened are related and involve the underlying SimulatePrincipalPolicy API. We will likely need to forward this issue to the IAM service team for review as they own and maintain the IAM APIs. But first I think we need a little more information to provide to them.
Could you share the specific command or code snippet, in addition to debug logs, to help us better understand this issue? If you're using the CLI command you can add --debug
to get the logs, or in Boto3 you can add boto3.set_stream_logger('')
. Please redact any sensitive information — we're just interested in seeing more details regarding the error you're encountering and how you arrived at it.
here is my sample code iam = boto3.client('iam', aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
policySourceArns = "arn:aws:iam::xxxxxxxx:role/cloudsoc_ct_role_effectivepermissions" retval = iam.simulate_principal_policy( PolicySourceArn = policySourceArns, ActionNames = ['rds:AddRoleToDBCluster', "iam:list*"] ) print("*" 10) print("All Permissions") print("*" 10) print([result["EvalActionName"] for result in retval["EvaluationResults"]])
permission = [result["EvalActionName"] for result in retval["EvaluationResults"] if result['EvalDecision'] == 'implicitDeny']
print("" 10) print("Denied Permissions") print("" * 10) print(permission)
Thanks @amitdwivedi-broadcom - and can you also share the logs, which you can get by adding boto3.set_stream_logger('')
to your code? As I mentioned you can redact any sensitive info, we just want to see the specific details of the error you are encountering.
/usr/bin/env /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 /Users/ad731973/.vscode/extensions/ms-python.debugpy-2024.4.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher 56132 -- /Users/ad731973/Desktop/myfiles/voice.py ad731973@GNX74DKQ9V myfiles % /usr/bin/env /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 /Users/ad731973/.vscode/extensions/ms-python.debugpy-2024.4.0-darwi n-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher 56132 -- /Users/ad731973/Desktop/myfiles/voice.py 2024-04-26 23:57:57,958 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'aws-global', 'UseDualStack': False, 'UseFIPS': False} 2024-04-26 23:57:57,959 botocore.regions [DEBUG] Endpoint provider result: https://iam.amazonaws.com 2024-04-26 23:57:57,959 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2024-04-26 23:57:57,959 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 'iam'} 2024-04-26 23:57:57,960 botocore.hooks [DEBUG] Event before-parameter-build.iam.SimulatePrincipalPolicy: calling handler <function generate_idempotent_uuid at 0x106057a30> 2024-04-26 23:57:57,960 botocore.hooks [DEBUG] Event before-call.iam.SimulatePrincipalPolicy: calling handler <function add_recursion_detection_header at 0x1060576d0> 2024-04-26 23:57:57,960 botocore.hooks [DEBUG] Event before-call.iam.SimulatePrincipalPolicy: calling handler <function inject_api_version_header_if_needed at 0x10606d2d0> 2024-04-26 23:57:57,960 botocore.endpoint [DEBUG] Making request for OperationModel(name=SimulatePrincipalPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.26.51 Python/3.10.7 Darwin/22.6.0 Botocore/1.29.51'}, 'body': {'Action': 'SimulatePrincipalPolicy', 'Version': '2010-05-08', 'PolicySourceArn': 'arn:aws:iam::xxxxxxxx:role/cloudsoc_ct_role', 'ActionNames.member.1': 's3:GetObject'}, 'url': 'https://iam.amazonaws.com/', 'context': {'client_region': 'aws-global', 'client_config': <botocore.config.Config object at 0x1068f8430>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'us-east-1', 'signing_name': 'iam'}}} 2024-04-26 23:57:57,961 botocore.hooks [DEBUG] Event request-created.iam.SimulatePrincipalPolicy: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1068f8280>> 2024-04-26 23:57:57,961 botocore.hooks [DEBUG] Event choose-signer.iam.SimulatePrincipalPolicy: calling handler <function set_operation_specific_signer at 0x106057910> 2024-04-26 23:57:57,961 botocore.auth [DEBUG] Calculating signature using v4 auth. 2024-04-26 23:57:57,961 botocore.auth [DEBUG] CanonicalRequest: POST /
content-type:application/x-www-form-urlencoded; charset=utf-8 host:iam.amazonaws.com x-amz-date:20240426T182757Z
content-type;host;x-amz-date
c6c1e810519cf75a76a4c52d5ac6dbe7b321fcbf166bd5ede7f0d10da451db25
2024-04-26 23:57:57,962 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20240426T182757Z
20240426/us-east-1/iam/aws4_request
44c611e414b2694fab95145dc526311d74ecde05c778e03248a2a2c93fbe2ad9
2024-04-26 23:57:57,962 botocore.auth [DEBUG] Signature:
66a9221a0983b6b511a95b6f3ca2926c0a564dee04fca793ee6edba0009904f4
2024-04-26 23:57:57,962 botocore.hooks [DEBUG] Event request-created.iam.SimulatePrincipalPolicy: calling handler <function add_retry_headers at 0x10606d990>
2024-04-26 23:57:57,962 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://iam.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'Boto3/1.26.51 Python/3.10.7 Darwin/22.6.0 Botocore/1.29.51', 'X-Amz-Date': b'20240426T182757Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=xxxxxxx/20240426/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=xxxxx', 'amz-sdk-invocation-id': b'xxxxxxxxxxx', 'amz-sdk-request': b'attempt=1', 'Content-Length': '161'}>
2024-04-26 23:57:57,964 botocore.httpsession [DEBUG] Certificate path: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certifi/cacert.pem
2024-04-26 23:57:57,964 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): iam.amazonaws.com:443
2024-04-26 23:57:59,095 urllib3.connectionpool [DEBUG] https://iam.amazonaws.com:443 "POST / HTTP/1.1" 200 765
2024-04-26 23:57:59,096 botocore.parsers [DEBUG] Response headers: {'Date': 'Fri, 26 Apr 2024 18:27:58 GMT', 'x-amzn-RequestId': '9a63dd7f-82ef-4026-9ace-93991e06b4b6', 'Content-Type': 'text/xml', 'Content-Length': '765'}
2024-04-26 23:57:59,096 botocore.parsers [DEBUG] Response body:
b'
All Permissions
['s3:GetObject']
Denied Permissions
['s3:GetObject']
Screenshot of Policy - GetObject should have gotten Allowed permission, but since this has regex "arn:aws:s3:::" its not working, if i changed that to only it works..
Hi @amitdwivedi-broadcom thanks for following up and providing more info. I'm wondering if the issue here is with your resource ARN format. Here is documentation related to that: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-arn-format.html. As described there, I think you might be looking for arn:aws:s3:::*
. Can you try that and let us know if it does what you're intending?
Also I noticed that the Boto3 version from your logs (1.26.51) is quite old. The latest version per the CHANGELOG is 1.34.94. We recommend using a more recent version for access to the latest updates/fixes.
i did the changes in policy and updated the boto3 but i still get the same result.
Boto log:: ad731973@GNX74DKQ9V myfiles % cd /Users/ad731973/Desktop/myfiles ; /usr/bin/env /Library/Frameworks/Python.framework/Versions/3.10/bin/python3 /Users/ad731973/.vscode/extens ions/ms-python.debugpy-2024.4.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher 54754 -- /Users/ad731973/Desktop/myfiles/voice.py 2024-04-30 16:59:09,723 botocore.hooks [DEBUG] Event before-parameter-build.iam.SimulatePrincipalPolicy: calling handler <function generate_idempotent_uuid at 0x104dbe680> 2024-04-30 16:59:09,724 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'aws-global', 'UseDualStack': False, 'UseFIPS': False} 2024-04-30 16:59:09,725 botocore.regions [DEBUG] Endpoint provider result: https://iam.amazonaws.com 2024-04-30 16:59:09,725 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None" 2024-04-30 16:59:09,725 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 'iam'} 2024-04-30 16:59:09,726 botocore.hooks [DEBUG] Event before-call.iam.SimulatePrincipalPolicy: calling handler <function add_recursion_detection_header at 0x104dbe320> 2024-04-30 16:59:09,726 botocore.hooks [DEBUG] Event before-call.iam.SimulatePrincipalPolicy: calling handler <function inject_api_version_header_if_needed at 0x104dbfeb0> 2024-04-30 16:59:09,726 botocore.endpoint [DEBUG] Making request for OperationModel(name=SimulatePrincipalPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.34.94 md/Botocore#1.34.94 ua/2.0 os/macos#22.6.0 md/arch#arm64 lang/python#3.10.7 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.94'}, 'body': {'Action': 'SimulatePrincipalPolicy', 'Version': '2010-05-08', 'PolicySourceArn': 'arn:aws:iam::xxxxxxx:role/cloudsoc_ct_role', 'ActionNames.member.1': 's3:GetObject'}, 'url': 'https://iam.amazonaws.com/', 'context': {'client_region': 'aws-global', 'client_config': <botocore.config.Config object at 0x1055fb1c0>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'us-east-1', 'signing_name': 'iam'}, 'endpoint_properties': {'authSchemes': [{'name': 'sigv4', 'signingName': 'iam', 'signingRegion': 'us-east-1'}]}}} 2024-04-30 16:59:09,726 botocore.hooks [DEBUG] Event request-created.iam.SimulatePrincipalPolicy: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1055faf80>> 2024-04-30 16:59:09,726 botocore.hooks [DEBUG] Event choose-signer.iam.SimulatePrincipalPolicy: calling handler <function set_operation_specific_signer at 0x104dbe560> 2024-04-30 16:59:09,727 botocore.auth [DEBUG] Calculating signature using v4 auth. 2024-04-30 16:59:09,727 botocore.auth [DEBUG] CanonicalRequest: POST /
content-type:application/x-www-form-urlencoded; charset=utf-8 host:iam.amazonaws.com x-amz-date:20240430T112909Z
content-type;host;x-amz-date
c6c1e810519cf75a76a4c52d5ac6dbe7b321fcbf166bd5ede7f0d10da451db25
2024-04-30 16:59:09,727 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20240430T112909Z
20240430/us-east-1/iam/aws4_request
a0e67961fa91a75f48dc192c37ce8a3130561d787c8d78afb20c52b0ebb20647
2024-04-30 16:59:09,727 botocore.auth [DEBUG] Signature:
0601eae453e4176121bb205b8fba3a187703ae8d5dbe3b5f4d5362f6fe350d9a
2024-04-30 16:59:09,727 botocore.hooks [DEBUG] Event request-created.iam.SimulatePrincipalPolicy: calling handler <function add_retry_headers at 0x104de05e0>
2024-04-30 16:59:09,728 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://iam.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'Boto3/1.34.94 md/Botocore#1.34.94 ua/2.0 os/macos#22.6.0 md/arch#arm64 lang/python#3.10.7 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.94', 'X-Amz-Date': b'20240430T112909Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=xxxxxxxxx/20240430/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=xxxxxxx', 'amz-sdk-invocation-id': b'cb6422a4-34b3-4e03-b379-412ba4e6d568', 'amz-sdk-request': b'attempt=1', 'Content-Length': '161'}>
2024-04-30 16:59:09,732 botocore.httpsession [DEBUG] Certificate path: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certifi/cacert.pem
2024-04-30 16:59:09,733 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): iam.amazonaws.com:443
2024-04-30 16:59:10,705 urllib3.connectionpool [DEBUG] https://iam.amazonaws.com:443 "POST / HTTP/1.1" 200 765
2024-04-30 16:59:10,706 botocore.parsers [DEBUG] Response headers: {'Date': 'Tue, 30 Apr 2024 11:29:10 GMT', 'x-amzn-RequestId': '61ac79ac-e02d-4153-b227-d9ce60b74d3e', 'Content-Type': 'text/xml', 'Content-Length': '765'}
2024-04-30 16:59:10,706 botocore.parsers [DEBUG] Response body:
b'
All Permissions
['s3:GetObject']
Denied Permissions
['s3:GetObject'] ad731973@GNX74DKQ9V myfiles %
Thanks for following up and sharing that info. I'm not sure what the issue is here but may be missing something. I'm going to reach out to the IAM team to try and get more information. Will also transfer this to our cross-SDK repository because as mentioned earlier this issue involves the API which is used across SDKs. (ref: P128030960)
sure thanks. Any update on this? its already 3 weeks
Hey @tim-finnigan we are waiting for an update on this, do let us know, I'm Priyanka from Amit's team and I will be following up on this.
Hey @tim-finnigan, do we have any update on this, we are waiting on this since more than a month now.
Hey @tim-finnigan , do we have any update on this?
Hey @tim-finnigan we are waiting for an update on this!
Hey @tim-finnigan we are waiting for an update on this!
@amitdwivedi-broadcom @PC-BRCM we are still waiting to hear back from the IAM team regarding this issue. It would be helpful if you could share the full updated IAM policy that you're using so that we can share that with the team.
Hi @tim-finnigan Do we have any ETA for the same? Its been more than a month we are waiting to hear on this? cloudsoc_policy.json PFA AWs CloudSoc Policy.
Thanks for your patience. Have you reviewed this documentation? https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html#policies-simulator-using-api
Specifically this part:
Please make sure you have the necessary IAM permissions. If still seeing an issue, please share your complete debug logs (with sensitive info redacted) by adding boto3.set_stream_logger('')
to highlight the issue.
Hey @tim-finnigan if you see Amit's policy in the decsription it already had these actions, which we tried and Amit has attached logs as well { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor14", "Action": [ "iam:GetContextKeysForCustomPolicy", "iam:GetContextKeysForPrincipalPolicy", "iam:SimulateCustomPolicy", "iam:SimulatePrincipalPolicy" ], "Effect": "Allow", "Resource": "*" } ] }
Hey @tim-finnigan here is my policy { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:GetAccelerateConfiguration", "s3:GetAnalyticsConfiguration", "s3:GetBucket", "s3:GetEncryptionConfiguration", "s3:GetInventoryConfiguration", "s3:GetLifecycleConfiguration", "s3:GetMetricsConfiguration", "s3:GetObjectAcl", "s3:GetObjectVersionAcl", "s3:GetReplicationConfiguration", "s3:ListAllMyBuckets", "cloudtrail:DescribeTrails", "cloudtrail:GetTrailStatus", "s3:GetObject", "s3:GetObjectAcl", "s3:PutObjectAcl", "s3:PutObjectVersionAcl", "s3:ListBucket", "s3:PutBucketNotification" ], "Resource": [ "arn:aws:s3:::" ], "Effect": "Allow", "Sid": "Stmt1396502506000" }, { "Action": [ "s3:GetBucketAcl", "s3:ListBucket", "s3:GetBucketLocation", "s3:PutBucketAcl", "s3:PutBucketNotification", "s3:ListAllMyBuckets" ], "Resource": [ "" ], "Effect": "Allow", "Sid": "Stmt1396502644000EDIT" }, { "Sid": "VisualEditor14", "Action": [ "iam:GetContextKeysForPrincipalPolicy", "iam:SimulatePrincipalPolicy" ], "Effect": "Allow", "Resource": "" } ] }
In the above policy, you can see for one block regular expression work for us, but for these action it is not working { "Action": [ "s3:GetBucketAcl", "s3:ListBucket", "s3:GetBucketLocation", "s3:PutBucketAcl", "s3:PutBucketNotification", "s3:ListAllMyBuckets" ], "Resource": [ "" ], "Effect": "Allow", "Sid": "Stmt1396502644000EDIT" },
and we have to use * inspite of RegEx, can you help us here?
@PC-BRCM The empty string here makes this policy invalid:
Thats a *
"Resource": [ " * " ],
That produces another error:
Can you try this policy? Note the fixes for both instances of Resource
: "arn:aws:s3:::*"
and "*"
:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetAccelerateConfiguration",
"s3:GetAnalyticsConfiguration",
"s3:GetBucket*",
"s3:GetEncryptionConfiguration",
"s3:GetInventoryConfiguration",
"s3:GetLifecycleConfiguration",
"s3:GetMetricsConfiguration",
"s3:GetObjectAcl",
"s3:GetObjectVersionAcl",
"s3:GetReplicationConfiguration",
"s3:ListAllMyBuckets",
"cloudtrail:DescribeTrails",
"cloudtrail:GetTrailStatus",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:PutObjectAcl",
"s3:PutObjectVersionAcl",
"s3:ListBucket",
"s3:PutBucketNotification"
],
"Resource": [
"arn:aws:s3:::*"
],
"Effect": "Allow",
"Sid": "Stmt1396502506000"
},
{
"Action": [
"s3:GetBucketAcl",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:PutBucketAcl",
"s3:PutBucketNotification",
"s3:ListAllMyBuckets"
],
"Resource": "*",
"Effect": "Allow",
"Sid": "Stmt1396502644000EDIT"
},
{
"Sid": "VisualEditor14",
"Action": [
"iam:GetContextKeysForPrincipalPolicy",
"iam:SimulatePrincipalPolicy"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
Let me share the policy json
So I have attached two policy, working where in for following action Resources [ * ] works for us but why is RegEx not working is the question ? "Action": [ "s3:GetBucketAcl", "s3:ListBucket", "s3:GetBucketLocation", "s3:PutBucketAcl", "s3:PutBucketNotification", "s3:ListAllMyBuckets" ],
Hey @tim-finnigan my query is not around the error in policy, the policy is working for us and dont have any syntax error, its the which is not coming in the comment here as its with escape character. My query is why the RegEx not working for the action i have mentioned and we have to use in resources inspite of arn:aws:s3:::*
Hi - @PC-BRCM the S3 User Guide notes here:
- The following ARN uses * to indicate all S3 buckets and objects.
arn:aws:s3:::*
So I would expect that to work. Can you provide the following for us to root cause:
boto3.set_stream_logger('')
to your script.The code/logs shared here earlier are incomplete.
@tim-finnigan can you also let us know if these permissions are deprecated now? SNS:Receive and "s3:GetPublicAccessBlock" -- these are invalid action, I can see these as invalid action while adding to policy, do we have new permissions for the same?
Hey @tim-finnigan did you get a chance to look into above query we asked?
Can you provide the info I requested above?
There is no SNS "Receive" API. For S3 you are probably looking for GetBucketPublicAccessBlock permissions as mentioned in https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetPublicAccessBlock.html
This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.
Describe the bug
Simulate policy API which helps us to check the effective permission for role is not working if the policy associated with role has regex like arn:aws::s3:* for resource or any such regex. It still gives that the permission is denied for actions which has resources associated with.
Problem in Python AWS SDK
API: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam/paginator/SimulatePrincipalPolicy.html
example:: { "Version": "2012-10-17", "Statement": [ { "Action": [ "iam:GetContextKeysForCustomPolicy", "iam:GetContextKeysForPrincipalPolicy", "iam:SimulateCustomPolicy", "iam:SimulatePrincipalPolicy" ], "Effect": "Allow", "Resource": "arn:aws::s3:*" } ] }
Expected Behavior
For a given role and the policy mentioned above we should get allowed effect for actions mentioned in the policy, however we get denied as effect. if we change "Resource": "arn:aws::s3:" to "Resource": "" this works as expected
Current Behavior
For a given role and the policy mentioned above we should get allowed effect for actions mentioned in the policy, however we get denied as effect. if we change "Resource": "arn:aws::s3:" to "Resource": "" this works as expected
Reproduction Steps
Create a policy and a role The policy should have regex in Resource field Write a POC using simulatePrinciplePolicy using this role and policy you created
Possible Solution
Ideally "arn:aws::s3:" should be treated as and it should give the same result
Additional Information/Context
No response
CLI version used
na
Environment details (OS name and version, etc.)
Ubuntu