search

noqdev / iambic

IAMbic is Version-Control for IAM. It centralizes and simplifies cloud access and permissions. It maintains an eventually consistent, human-readable, bi-directional representation of IAM in Git.
https://iambic.org
Apache License 2.0
281 stars 26 forks source link

ResourceNotFoundException in import resources #587

Closed rodolphoescobar closed 1 year ago

rodolphoescobar commented 1 year ago

Describe the bug iambic import resources not work with AWS Organizations

To Reproduce Steps to reproduce the behavior:

Run iambic import with AWS Organizations

Error log

error=ResourceNotFoundException('An error occurred (ResourceNotFoundException) when calling the DescribeGroup operation: GROUP not found.') exception=ResourceNotFoundException stacktrace= File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/app/iambic/main.py", line 542, in cli() File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, *kwargs) File "/app/iambic/main.py", line 447, in import_ asyncio.run(config.run_import(exe_message, repo_dir)) File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/app/iambic/config/dynamic_config.py", line 268, in run_import await asyncio.gather(tasks) File "/app/iambic/plugins/v0_1_0/aws/handlers.py", line 462, in import_aws_resources await asyncio.gather(tasks) File "/app/iambic/plugins/v0_1_0/aws/handlers.py", line 380, in import_identity_center_resources await import_service_resources( File "/app/iambic/plugins/v0_1_0/aws/handlers.py", line 336, in import_service_resources await asyncio.gather(tasks) File "/app/iambic/plugins/v0_1_0/aws/identity_center/permission_set/template_generation.py", line 520, in collect_aws_permission_sets all_permission_sets = await generate_permission_set_resource_file_semaphore.process( File "/app/iambic/core/utils.py", line 227, in process return await asyncio.gather( File "/app/iambic/core/utils.py", line 222, in handle_message return await self.callback_function(**kwargs) File "/app/iambic/plugins/v0_1_0/aws/identity_center/permission_set/template_generation.py", line 123, in generate_permission_set_resource_file permission_set["assignments"] = await get_permission_set_users_and_groups( File "/app/iambic/plugins/v0_1_0/aws/identity_center/permission_set/utils.py", line 152, in get_permission_set_users_and_groups info = wrap_identity_store_client.describe_group(aa["PrincipalId"]) File "/app/iambic/plugins/v0_1_0/aws/identity_center/permission_set/utils.py", line 54, in describe_group return self.boto3_identity_center_store_client.describe_group( File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 534, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 976, in _make_api_call raise error_class(parsed_response, operation_name)

castrapel commented 1 year ago

Thanks for reporting @rodolphoescobar . How are you sourcing groups for Identity Center? Through AD sync? Are you using SCIM? Are you still experiencing this with the latest version of IAMbic (We've pushed some identity center fixes recently)

Also if you're not in Slack, we'd be delighted to have you there: https://communityinviter.com/apps/noqcommunity/noq

rodolphoescobar commented 1 year ago

Hi @castrapel !

We use synchronization with Azure AD via SCIM.

I tested it on the latest version 0.11.52 locally and docker latest image.

Thanks!

smoy commented 1 year ago

hi @rodolphoescobar , to narrow the condition to reproduce the bug, we like to collect the pattern of username and group.

For example,

we have received another user report an issue like user in LAST.FIRST@LOCAL_DOMAIN (LOCAL DOMAIN is not a fully qualified domain, like example.com)

we are interest to know whether your group pattern to reproduce. (if its something simple like engineering or more complex like engineering@LOCAL_DOMAIN.

meanwhile, I will enhance the error message to capture the resource that cannot be resolved.

(we are working with another user that has AD + IdentityCenter not able to resolve resources because of boto3 and identity center service interaction. we are still investigating what kind of workaround is available)

smoy commented 1 year ago

@rodolphoescobar we have release version 0.11.56 that handles the AD + IdentityCenter issue on not able to describe_user or describe_group. This will record the PrincipalID just like the control plane response.

That should unlock the import steps. We are still investigating other AD issue on https://github.com/noqdev/iambic/issues/557

rodolphoescobar commented 1 year ago

@smoy

My group list:

    "CL-APP-AWS-BILLING",
    "CL-SQUAD-MOBILE-PLATFORM",
    "CL-SQUAD-CROSS-FINANCE-ERP",
    "CL-APP-AWS-MATERA",
    "CL-APP-AWS-INTEGRATIONS",
    "CL-SQUAD-CROSS-FINANCE-PLATFORM",
    "CL-SQUAD-DEVOIP",
    "CL-SQUAD-DEVSECOPS",
    "CL-APP-AWS-DEVELOPERS",
    "CL-APP-AWS-BCP",
    "CL-APP-AWS-DEVELOPERS-ZAP",
    "CL-SQUAD-DATA-ENGINEERING-ADM",
    "CL-SQUAD-REGULATORY-DATA",
    "CL-SQUAD-DEV-TOOLS",
    "CL-SQUAD-ITOPS",
    "CL-SQUAD-SERVICEDESK",
    "CL-SQUAD-ZAP-VIVAREAL-MATCHING",
    "CL-SQUAD-DATASCIENCE-TEAM",
    "CL-APP-AWS-DATTOS",
    "CL-SQUAD-FINOPS",
    "CL-APP-AWS-INHOUSE",
    "CL-SQUAD-MATCHING",
    "CL-APP-AWS-RECURSOS-HUMANOS",
    "CL-APP-AWS-WALLET-CORE",
    "CL-SQUAD-USER-NOTIFICATION",
    "CL-SQUAD-FOUNDATION",
    "CL-SQUAD-DATA-ENGINEERING",
    "CL-SQUAD-USER-MODERATION",
    "CL-APP-AWS-DEVELOPERS-OLX",
    "CL-SQUAD-FINANCEIRO-3",
    "CL-SQUAD-OBSERVABILITY",
    "CL-SQUAD-CYBERSEC"
smoy commented 1 year ago

Yea, this seems like it's due to describe_group not able to query for the data. Can you confirm this by using AWS CLI to perform the following:

aws identitystore describe-group --identity-store-id YOUR_UNIQUE_ID_STORE_ID --group-id GROUP_ID

you can find YOUR_UNIQUE_ID_STORE_ID from IdentityCenter dashboard on AWS console on the settings summary

You can find GROUP_ID from one of the group in Groups on AWS console.

docs: identitystore

If you encounter error from aws-cli, that will confirm the theory that boto3 API is not sufficient to handle your AD setup in identity center, and we will need to rely on the recent workaround from https://github.com/noqdev/iambic/pull/591