aws-solutions / account-assessment-for-aws-organizations

Account Assessment for AWS Organizations programmatically scans all AWS accounts in an AWS Organization for identity-based and resource-based policies with Organization-based conditions.
Apache License 2.0
25 stars 10 forks source link

Queries to services do not return all resources because paging is not being used #8

Closed vinelias closed 1 year ago

vinelias commented 1 year ago

Describe the bug When working on accounts with large numbers of resources in a given service, only a portion of the resources are returned in the query. For example, if we have more than 100 IAM roles, the query will return only the first 100, as this is the default limit of the list_roles method page of the iam service in boto3. (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam/client/list_roles.html). This goes for other resources like IAM Policies, secrets in Secrets Manager, KMS Keys, SNS Topics, etc...

To Reproduce Take an account over 100 IAM roles and start a Resource-Based Policies scan. You will only see the first 100 IAM roles.

Expected behavior View all IAM roles in the account

Please complete the following information about the solution:

To get the version of the solution, you can look at the description of the created CloudFormation stack.

For example, "(SO0217) - The AWS CloudFormation hub template for deployment of the Account Assessment for AWS Organisations, Version: v1.0.0".

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context Add any other context about the problem here. I collected the information I needed using a personal script and applied paging whenever the resource allowed. e.g:

#Get IAM roles
...
    resource_list = []
    paginator = client.get_paginator('list_roles')
    page_iterator = paginator.paginate()
    for page in page_iterator:
      for item in page.get('Roles'):
        if item is not None:
          resource_list.append(item)
....
if resource_list is not None:
    try:
      for resource in resource_list:
....
vinelias commented 1 year ago

My bad! This issue was opened erroneously. The solution is working as expected, the test was done wrong.