Closed oleksandr-onsha closed 4 years ago
Added an error to retry'able and apply retry to the function that performs create_function call. The log of deploy with applied fix:
2020-03-18 11:43:49,646 [INFO] sdct.syndicate.core.resources.iam_resource,171 Created IAM role sdct-telegram-bot-api-role-smpl. 2020-03-18 11:43:49,646 [INFO] sdct.syndicate.core.build.deployment_processor,93 Processing lambda resources ... 2020-03-18 11:43:49,647 [DEBUG] sdct.syndicate.core.resources.lambda_resource,186 Creating lambda telegram-bot-api-function 2020-03-18 11:43:50,971 [DEBUG] sdct.syndicate.connection.helper,78 Retry on create_lambda. Error: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. 2020-03-18 11:43:50,972 [DEBUG] sdct.syndicate.connection.helper,80 Parameters: (<syndicate.connection.lambda_connection.LambdaConnection object at 0x10aa79710>,), {'lambda_name': 'telegram-bot-api-function', 'func_name': 'handler.lambda_handler', 'role': 'arn:aws:iam::854412417832:role/sdct-telegram-bot-api-role-smpl', 'runtime': 'python3.7', 'memory': 128, 'timeout': 300, 's3_bucket': 'aws-syndicate-demo-bucket', 's3_key': 'yuliia_1703_1/telegram-bot-api-function-1.0.zip', 'env_vars': {'region': 'eu-west-1'}, 'vpc_sub_nets': None, 'vpc_security_group': None, 'dl_target_arn': None, 'tracing_mode': None, 'publish_version': False, 'layers': []} 2020-03-18 11:43:50,981 [DEBUG] sdct.syndicate.connection.helper,83 Traceback: Traceback (most recent call last): File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/helper.py", line 72, in wrapper return handler_func(args, kwargs) File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/lambda_connection.py", line 96, in create_lambda return self.client.create_function(params) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. 2020-03-18 11:43:52,308 [DEBUG] sdct.syndicate.connection.helper,78 Retry on create_lambda. Error: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. 2020-03-18 11:43:52,309 [DEBUG] sdct.syndicate.connection.helper,80 Parameters: (<syndicate.connection.lambda_connection.LambdaConnection object at 0x10aa79710>,), {'lambda_name': 'telegram-bot-api-function', 'func_name': 'handler.lambda_handler', 'role': 'arn:aws:iam::854412417832:role/sdct-telegram-bot-api-role-smpl', 'runtime': 'python3.7', 'memory': 128, 'timeout': 300, 's3_bucket': 'aws-syndicate-demo-bucket', 's3_key': 'yuliia_1703_1/telegram-bot-api-function-1.0.zip', 'env_vars': {'region': 'eu-west-1'}, 'vpc_sub_nets': None, 'vpc_security_group': None, 'dl_target_arn': None, 'tracing_mode': None, 'publish_version': False, 'layers': []} 2020-03-18 11:43:52,309 [DEBUG] sdct.syndicate.connection.helper,83 Traceback: Traceback (most recent call last): File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/helper.py", line 72, in wrapper return handler_func(args, kwargs) File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/lambda_connection.py", line 96, in create_lambda return self.client.create_function(params) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. 2020-03-18 11:43:56,670 [DEBUG] sdct.syndicate.core.resources.lambda_resource,249 Lambda created telegram-bot-api-function 2020-03-18 11:44:06,773 [DEBUG] sdct.syndicate.core.resources.lambda_resource,268 arn value: arn:aws:lambda:eu-west-1:854412417832:function:telegram-bot-api-function 2020-03-18 11:44:06,991 [INFO] sdct.syndicate.core.build.deployment_processor,294 AWS resources were deployed successfully 2020-03-18 11:44:06,992 [INFO] sdct.syndicate.core.build.deployment_processor,297 Going to apply dynamic changes 2020-03-18 11:44:06,992 [INFO] sdct.syndicate.core.build.deployment_processor,299 Dynamic changes were applied successfully 2020-03-18 11:44:06,992 [INFO] sdct.syndicate.core.build.deployment_processor,301 Going to create deploy output 2020-03-18 11:44:07,610 [INFO] sdct.syndicate.core.build.bundle_processor,62 Output file with name yuliia_1703_1/outputs/sdct.json has been created 2020-03-18 11:44:07,610 [INFO] sdct.syndicate.core.build.deployment_processor,307 Deploy output for sdct was created. 2020-03-18 11:44:07,610 [INFO] sdct.syndicate.core.helper,214 Stage deploy, elapsed time: 0:00:22.185713
The lambda was created after 2 retries - such time IAM and Lambda services need to sync latest data between each other.
Fix is available since v 0.8.5
Describe the bug An error occurred while deploying an application that consists of the following resources: 1 IAM Policy 1 IAM Role 1 Lambda function The priority of creation is the following: policy, role, lambda; aws-syndicate failed with the botocore.errorfactory.InvalidParameterValueException on lambda creation; The error is floating and depends on the speed of 'deploy' command execution. The reason is that aws-syndicate creates lambda right after the role - to few time passes after role creation. AWS IAM does not have enough time to populate the latest changes and AWS Lambda service could not get info about the just created role.
To Reproduce
Expected behavior Sample application should deploy correctly.
Traceback
Additional context Contacted AWS Support to clarify is there a way to check if just created role is available to Lambda. Recommended way to solve the issue: add time.sleep() after creation of IAM role to allow take some time for AWS IAM service to perform sync between services.