search

awslabs / aws-deployment-framework

The AWS Deployment Framework (ADF) is an extensive and flexible framework to manage and deploy resources across multiple AWS accounts and regions based on AWS Organizations.
Apache License 2.0
662 stars 226 forks source link

Using wave deployment functionality #427

Closed hynynen closed 2 years ago

hynynen commented 2 years ago

Hi!

I'm testing waves described in the documentation: https://github.com/awslabs/aws-deployment-framework/blob/master/docs/user-guide.md#providers

Iäm using the exact syntax described in the example, but I get error:

2022-01-11 13:03:25,598 \| INFO \| deployment_map \| Loading deployment_map file deployment_maps/inactive-iam-users.yml \| (deployment_map.py:65)
--
97 | 2022-01-11 13:03:25,601 \| ERROR \| deployment_map \| Key 'pipelines' error:
98 | Or({'name': And(<class 'str'>, <built-in function len>), 'default_providers': {'source': And({'provider': Or('codecommit', 'github', 's3'), 'properties': <class 'dict'>}, <function <lambda> at 0x7fe80f7b6280>), Optional('build'): And({Optional('provider'): Or('codebuild', 'jenkins'), Optional('enabled'): <class 'bool'>, Optional('properties'): <class 'dict'>}, <function <lambda> at 0x7fe80f7b6310>), Optional('deploy'): And({'provider': Or('cloudformation', 's3', 'codedeploy', 'lambda', 'service_catalog', 'codebuild'), Optional('enabled'): <class 'bool'>, Optional('properties'): <class 'dict'>}, <function <lambda> at 0x7fe80f7b63a0>)}, Optional('params'): {Optional('notification_endpoint'): <class 'str'>, Optional('schedule'): <class 'str'>, Optional('restart_execution_on_update'): <class 'bool'>}, Optional('tags'): <class 'dict'>, Optional('targets'): [Or(<class 'str'>, <class 'int'>, {Optional('path'): Or(<class 'str'>, <class 'int'>, [Or(<class 'str'>, <class 'int'>)]), Optional('tags'): {And(<class 'str'>, Regex('\\A.{1,128}\\Z')): And(<class 'str'>, Regex('\\A.{0,256}\\Z'))}, Optional('target'): Or(<class 'str'>, <class 'int'>, [Or(<class 'str'>, <class 'int'>)]), Optional('name'): <class 'str'>, Optional('provider'): Or('lambda', 's3', 'codedeploy', 'cloudformation', 'service_catalog', 'approval', 'codebuild', 'jenkins'), Optional('properties'): Or({Optional('image'): Or(<class 'str'>, {'repository_arn': <class 'str'>, Optional('tags'): <class 'dict'>}), Optional('size'): Or('small', 'medium', 'large'), Optional('spec_filename'): <class 'str'>, Optional('environment_variables'): {Optional(<class 'str'>): Or(<class 'str'>, <class 'bool'>, <class 'int'>, <class 'object'>)}, Optional('role'): <class 'str'>, Optional('timeout'): <class 'int'>, Optional('privileged'): <class 'bool'>, Optional('spec_inline'): <class 'object'>}, {Optional('project_name'): <class 'str'>, Optional('server_url'): <class 'str'>, Optional('provider_name'): <class 'str'>}, {Optional('stack_name'): <class 'str'>, Optional('template_filename'): <class 'str'>, Optional('root_dir'): <class 'str'>, Optional('role'): <class 'str'>, Optional('action'): Or('CHANGE_SET_EXECUTE', 'CHANGE_SET_REPLACE', 'CREATE_UPDATE', 'DELETE_ONLY', 'REPLACE_ON_FAILURE', 'change_set_execute', 'change_set_replace', 'create_update', 'delete_only', 'replace_on_failure'), Optional('outputs'): <class 'str'>, Optional('change_set_approval'): <class 'bool'>, Optional('param_overrides'): [{'inputs': <class 'str'>, 'param': <class 'str'>, 'key_name': <class 'str'>}]}, {'application_name': <class 'str'>, 'deployment_group_name': <class 'str'>, Optional('role'): <class 'str'>}, {'bucket_name': <class 'str'>, 'object_key': <class 'str'>, Optional('extract'): <class 'bool'>, Optional('role'): <class 'str'>}, {'product_id': <class 'str'>, Optional('configuration_file_path'): <class 'str'>}, {'function_name': <class 'str'>, Optional('input'): Or(<class 'str'>, <class 'object'>), Optional('role'): <class 'str'>}, {Optional('message'): <class 'str'>, Optional('notification_endpoint'): <class 'str'>, Optional('sns_topic_arn'): <class 'str'>}), Optional('regions'): Or(<class 'str'>, <class 'list'>)}, [Or(<class 'str'>, <class 'int'>)])], Optional('regions'): Or(<class 'str'>, <class 'list'>), Optional('completion_trigger'): {'pipelines': [<class 'str'>]}}) did not validate {'name': 'inactive-iam-users', 'default_providers': {'source': {'provider': 'codecommit', 'properties': {'account_id': 177896491379}}}, 'params': {'restart_execution_on_update': True}, 'targets': [{'path': '/new-accounts-adf', 'regions': 'eu-west-1'}, {'path': '/new-accounts-adf-2', 'regions': 'eu-west-1', 'wave': {'size': 30}}, {'path': '/Customer', 'regions': 'eu-west-1'}, {'path': '/dish/new', 'regions': 'eu-west-1'}]}
99 | Key 'targets' error:
100 | Or(Or(<class 'str'>, <class 'int'>, {Optional('path'): Or(<class 'str'>, <class 'int'>, [Or(<class 'str'>, <class 'int'>)]), Optional('tags'): {And(<class 'str'>, Regex('\\A.{1,128}\\Z')): And(<class 'str'>, Regex('\\A.{0,256}\\Z'))}, Optional('target'): Or(<class 'str'>, <class 'int'>, [Or(<class 'str'>, <class 'int'>)]), Optional('name'): <class 'str'>, Optional('provider'): Or('lambda', 's3', 'codedeploy', 'cloudformation', 'service_catalog', 'approval', 'codebuild', 'jenkins'), Optional('properties'): Or({Optional('image'): Or(<class 'str'>, {'repository_arn': <class 'str'>, Optional('tags'): <class 'dict'>}), Optional('size'): Or('small', 'medium', 'large'), Optional('spec_filename'): <class 'str'>, Optional('environment_variables'): {Optional(<class 'str'>): Or(<class 'str'>, <class 'bool'>, <class 'int'>, <class 'object'>)}, Optional('role'): <class 'str'>, Optional('timeout'): <class 'int'>, Optional('privileged'): <class 'bool'>, Optional('spec_inline'): <class 'object'>}, {Optional('project_name'): <class 'str'>, Optional('server_url'): <class 'str'>, Optional('provider_name'): <class 'str'>}, {Optional('stack_name'): <class 'str'>, Optional('template_filename'): <class 'str'>, Optional('root_dir'): <class 'str'>, Optional('role'): <class 'str'>, Optional('action'): Or('CHANGE_SET_EXECUTE', 'CHANGE_SET_REPLACE', 'CREATE_UPDATE', 'DELETE_ONLY', 'REPLACE_ON_FAILURE', 'change_set_execute', 'change_set_replace', 'create_update', 'delete_only', 'replace_on_failure'), Optional('outputs'): <class 'str'>, Optional('change_set_approval'): <class 'bool'>, Optional('param_overrides'): [{'inputs': <class 'str'>, 'param': <class 'str'>, 'key_name': <class 'str'>}]}, {'application_name': <class 'str'>, 'deployment_group_name': <class 'str'>, Optional('role'): <class 'str'>}, {'bucket_name': <class 'str'>, 'object_key': <class 'str'>, Optional('extract'): <class 'bool'>, Optional('role'): <class 'str'>}, {'product_id': <class 'str'>, Optional('configuration_file_path'): <class 'str'>}, {'function_name': <class 'str'>, Optional('input'): Or(<class 'str'>, <class 'object'>), Optional('role'): <class 'str'>}, {Optional('message'): <class 'str'>, Optional('notification_endpoint'): <class 'str'>, Optional('sns_topic_arn'): <class 'str'>}), Optional('regions'): Or(<class 'str'>, <class 'list'>)}, [Or(<class 'str'>, <class 'int'>)])) did not validate {'path': '/new-accounts-adf-2', 'regions': 'eu-west-1', 'wave': {'size': 30}}
101 | Or(<class 'str'>, <class 'int'>, {Optional('path'): Or(<class 'str'>, <class 'int'>, [Or(<class 'str'>, <class 'int'>)]), Optional('tags'): {And(<class 'str'>, Regex('\\A.{1,128}\\Z')): And(<class 'str'>, Regex('\\A.{0,256}\\Z'))}, Optional('target'): Or(<class 'str'>, <class 'int'>, [Or(<class 'str'>, <class 'int'>)]), Optional('name'): <class 'str'>, Optional('provider'): Or('lambda', 's3', 'codedeploy', 'cloudformation', 'service_catalog', 'approval', 'codebuild', 'jenkins'), Optional('properties'): Or({Optional('image'): Or(<class 'str'>, {'repository_arn': <class 'str'>, Optional('tags'): <class 'dict'>}), Optional('size'): Or('small', 'medium', 'large'), Optional('spec_filename'): <class 'str'>, Optional('environment_variables'): {Optional(<class 'str'>): Or(<class 'str'>, <class 'bool'>, <class 'int'>, <class 'object'>)}, Optional('role'): <class 'str'>, Optional('timeout'): <class 'int'>, Optional('privileged'): <class 'bool'>, Optional('spec_inline'): <class 'object'>}, {Optional('project_name'): <class 'str'>, Optional('server_url'): <class 'str'>, Optional('provider_name'): <class 'str'>}, {Optional('stack_name'): <class 'str'>, Optional('template_filename'): <class 'str'>, Optional('root_dir'): <class 'str'>, Optional('role'): <class 'str'>, Optional('action'): Or('CHANGE_SET_EXECUTE', 'CHANGE_SET_REPLACE', 'CREATE_UPDATE', 'DELETE_ONLY', 'REPLACE_ON_FAILURE', 'change_set_execute', 'change_set_replace', 'create_update', 'delete_only', 'replace_on_failure'), Optional('outputs'): <class 'str'>, Optional('change_set_approval'): <class 'bool'>, Optional('param_overrides'): [{'inputs': <class 'str'>, 'param': <class 'str'>, 'key_name': <class 'str'>}]}, {'application_name': <class 'str'>, 'deployment_group_name': <class 'str'>, Optional('role'): <class 'str'>}, {'bucket_name': <class 'str'>, 'object_key': <class 'str'>, Optional('extract'): <class 'bool'>, Optional('role'): <class 'str'>}, {'product_id': <class 'str'>, Optional('configuration_file_path'): <class 'str'>}, {'function_name': <class 'str'>, Optional('input'): Or(<class 'str'>, <class 'object'>), Optional('role'): <class 'str'>}, {Optional('message'): <class 'str'>, Optional('notification_endpoint'): <class 'str'>, Optional('sns_topic_arn'): <class 'str'>}), Optional('regions'): Or(<class 'str'>, <class 'list'>)}, [Or(<class 'str'>, <class 'int'>)]) did not validate {'path': '/new-accounts-adf-2', 'regions': 'eu-west-1', 'wave': {'size': 30}}
102 | {'path': '/new-accounts-adf-2', 'regions': 'eu-west-1', 'wave': {'size': 30}} should be instance of 'str'
103 | {'path': '/new-accounts-adf-2', 'regions': 'eu-west-1', 'wave': {'size': 30}} should be instance of 'int'
104 | Wrong key 'wave' in {'path': '/new-accounts-adf-2', 'regions': 'eu-west-1', 'wave': {'size': 30}}
105 | {'path': '/new-accounts-adf-2', 'regions': 'eu-west-1', 'wave': {'size': 30}} should be instance of 'list' \| (deployment_map.py:73)

Also, the documentation states that "A new stage is introduced for every 50 accounts/region deployments by default" which implies that I wouldn't even need to use this parameter as ADF should be doing the splitting automatically when I have more than 50 accounts in the OU.

sbkok commented 2 years ago

Hi @hynynen,

We are working on a new release of ADF that will include this functionality. The wave concept has not been released yet. It will be released as part of v3.2.0.

On the current master branch, there are a few issues with the wave concept that I am fixing too. A merge request will follow to address those issues. So I am afraid you have to wait a bit longer before you can use this feature. Apologies for the delay.

Best regards, Simon

hynynen commented 2 years ago

Thank you for the answer! I was wondering that there is something like this in it. I was just confused that the documentation already speaks about this subject, I had to try it.

sbkok commented 1 year ago

Thank you for your patience. I am happy to inform you that this feature has been released as part of v3.2.0 just now. Please open a new issue if you are experiencing any issues related to this feature.