snowflakedb / SnowAlert

Security Analytics Using The Snowflake Data Warehouse
Apache License 2.0
182 stars 57 forks source link

[DC] Collect AWSIC IAM managed/inline policy data for users/groups/roles #550

Closed stevenliu-czi closed 3 years ago

stevenliu-czi commented 3 years ago

This has not been tested and was written based off pattern matching on previous examples within the files.

stevenliu-czi commented 3 years ago

None of the commits have been tested and the new code changes were written based off pattern matching on previous examples within the files.

stevenliu-czi commented 3 years ago

[Updated comment for new commit] Added syntax fixes. Still not tested.

sfc-gh-afedorov commented 3 years ago

tested on one of our internal accounts

(ve) {prod}[AAA] src % SA_ENV=dev ipython --pdb -- runners/connectors_runner.py AWS_COLLECT_ORGANIZATIONS_LIST_ACCOUNTS_CONNECTION --run-now --dryrun False --debug --org-account-ids 'CCC' --collect-apis 'iam.list_groups,iam.list_users,iam.list_roles'
/usr/local/lib/python3.8/site-packages/IPython/core/interactiveshell.py:936: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
  warn("Attempting to work in a virtualenv. If you encounter problems, please "
[57339] -- START DC AWS_COLLECT_ORGANIZATIONS_LIST_ACCOUNTS_CONNECTION --
[57339] RUN metadata recorded.
[57339] botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the ListAccounts operation: You don't have permissions to access this resource.
[57339] inserting 1 values into data.AWS_COLLECT_ORGANIZATIONS_LIST_ACCOUNTS_CONNECTION
[57339] progress: starting 3, queued 0
[57339] inserting 2 values into data.aws_collect_iam_list_groups
[57339] finished iam.list_groups {'number of rows inserted': 2}
[57339] inserting 23 values into data.aws_collect_iam_list_users
[57339] finished iam.list_users {'number of rows inserted': 23}
[57339] inserting 134 values into data.aws_collect_iam_list_roles
[57339] finished iam.list_roles {'number of rows inserted': 134}
[57339] progress: starting 410, queued 0
[57339] botocore.errorfactory.NoSuchEntityException: An error occurred (NoSuchEntity) when calling the GetLoginProfile operation: Login Profile for User BBB cannot be found.
[57339] botocore.errorfactory.NoSuchEntityException: An error occurred (NoSuchEntity) when calling the GetLoginProfile operation: Login Profile for User BBB cannot be found.
[57339] botocore.errorfactory.NoSuchEntityException: An error occurred (NoSuchEntity) when calling the GetLoginProfile operation: Login Profile for User BBB cannot be found.
[57339] botocore.errorfactory.NoSuchEntityException: An error occurred (NoSuchEntity) when calling the GetLoginProfile operation: Login Profile for User BBB cannot be found.
[57339] botocore.errorfactory.NoSuchEntityException: An error occurred (NoSuchEntity) when calling the GetLoginProfile operation: Login Profile for User BBB cannot be found.
[57339] botocore.errorfactory.NoSuchEntityException: An error occurred (NoSuchEntity) when calling the GetLoginProfile operation: Login Profile for User BBB cannot be found.
[57339] inserting 2 values into data.aws_collect_iam_list_group_policies
[57339] finished iam.list_group_policies {'number of rows inserted': 2}
[57339] inserting 3 values into data.aws_collect_iam_list_attached_group_policies
[57339] finished iam.list_attached_group_policies {'number of rows inserted': 3}
[57339] inserting 25 values into data.aws_collect_iam_list_access_keys
[57339] finished iam.list_access_keys {'number of rows inserted': 25}
[57339] inserting 23 values into data.aws_collect_iam_list_mfa_devices
[57339] finished iam.list_mfa_devices {'number of rows inserted': 23}
[57339] inserting 23 values into data.aws_collect_iam_list_user_policies
[57339] finished iam.list_user_policies {'number of rows inserted': 23}
[57339] inserting 28 values into data.aws_collect_iam_list_groups_for_user
[57339] finished iam.list_groups_for_user {'number of rows inserted': 28}
[57339] inserting 24 values into data.aws_collect_iam_list_attached_user_policies
[57339] finished iam.list_attached_user_policies {'number of rows inserted': 24}
[57339] inserting 23 values into data.aws_collect_iam_get_login_profile
[57339] finished iam.get_login_profile {'number of rows inserted': 23}
[57339] inserting 174 values into data.aws_collect_iam_list_attached_role_policies
[57339] finished iam.list_attached_role_policies {'number of rows inserted': 174}
[57339] inserting 140 values into data.aws_collect_iam_list_role_policies
[57339] finished iam.list_role_policies {'number of rows inserted': 140}
[57339] progress: starting 52, queued 0
[57339] inserting 49 values into data.aws_collect_iam_get_role_policy
[57339] finished iam.get_role_policy {'number of rows inserted': 49}
[57339] inserting 2 values into data.aws_collect_iam_get_user_policy
[57339] finished iam.get_user_policy {'number of rows inserted': 2}
[57339] inserting 1 values into data.aws_collect_iam_get_group_policy
[57339] finished iam.get_group_policy {'number of rows inserted': 1}
[57339] RUN metadata recorded.
[57339] -- END DC AWS_COLLECT_ORGANIZATIONS_LIST_ACCOUNTS_CONNECTION --
SA_ENV=dev ipython --pdb -- runners/connectors_runner.py  --run-now --dryrun   11.39s user 1.07s system 24% cpu 50.431 total