search

epam / aws-syndicate

Syndicate deployment framework
Apache License 2.0
57 stars 11 forks source link

[BUG] Error while creating lambda function right after creation of role for the lambda #63

Closed oleksandr-onsha closed 4 years ago

oleksandr-onsha commented 4 years ago

Describe the bug An error occurred while deploying an application that consists of the following resources: 1 IAM Policy 1 IAM Role 1 Lambda function The priority of creation is the following: policy, role, lambda; aws-syndicate failed with the botocore.errorfactory.InvalidParameterValueException on lambda creation; The error is floating and depends on the speed of 'deploy' command execution. The reason is that aws-syndicate creates lambda right after the role - to few time passes after role creation. AWS IAM does not have enough time to populate the latest changes and AWS Lambda service could not get info about the just created role.

To Reproduce

  1. Remove all resources from sample/sample-python-project except one 1 policy and 1 role and 1 lambda function;
  2. Configure sdct.conf and sdct_aliases.conf;
  3. Deploy sample app;

Expected behavior Sample application should deploy correctly.

Traceback

(.venv) ➜ control-money-python-bot git:(aws-syndicate-restructuring) ✗ syndicate deploy --bundle_name yuliia_1703 --deploy_name bot Path to sdct.conf: /Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/config Command deploy backend Deploy name: bot build_meta.json file was loaded successfully Going to deploy AWS resources Processing iam_policy resources ... Created IAM policy sdct-LambdaBasicExecution-smpl. Processing iam_role resources ... Created IAM role sdct-telegram-bot-api-role-smpl. Processing lambda resources ... Error occurred while lambda resource creating: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. Traceback (most recent call last): File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/core/build/deployment_processor.py", line 95, in _process_resources response = func(args) File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/core/resources/lambda_resource.py", line 119, in create_lambda return create_pool(_create_lambda_from_meta, args) File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/core/helper.py", line 51, in create_pool result = future.result() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 425, in result return self.get_result() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in get_result raise self._exception File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(self.args, self.kwargs) File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/core/helper.py", line 69, in wrapper return handler_func(kwargs[0]) File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/core/resources/lambda_resource.py", line 246, in _create_lambda_from_meta layers=lambda_layers_arns File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/helper.py", line 85, in wrapper raise e File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/helper.py", line 71, in wrapper return handler_func(args, kwargs) File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/lambda_connection.py", line 96, in create_lambda return self.client.create_function(params) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. Error occurred: 'NoneType' object is not subscriptable

Additional context Contacted AWS Support to clarify is there a way to check if just created role is available to Lambda. Recommended way to solve the issue: add time.sleep() after creation of IAM role to allow take some time for AWS IAM service to perform sync between services.

oleksandr-onsha commented 4 years ago

Added an error to retry'able and apply retry to the function that performs create_function call. The log of deploy with applied fix:

2020-03-18 11:43:49,646 [INFO] sdct.syndicate.core.resources.iam_resource,171 Created IAM role sdct-telegram-bot-api-role-smpl. 2020-03-18 11:43:49,646 [INFO] sdct.syndicate.core.build.deployment_processor,93 Processing lambda resources ... 2020-03-18 11:43:49,647 [DEBUG] sdct.syndicate.core.resources.lambda_resource,186 Creating lambda telegram-bot-api-function 2020-03-18 11:43:50,971 [DEBUG] sdct.syndicate.connection.helper,78 Retry on create_lambda. Error: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. 2020-03-18 11:43:50,972 [DEBUG] sdct.syndicate.connection.helper,80 Parameters: (<syndicate.connection.lambda_connection.LambdaConnection object at 0x10aa79710>,), {'lambda_name': 'telegram-bot-api-function', 'func_name': 'handler.lambda_handler', 'role': 'arn:aws:iam::854412417832:role/sdct-telegram-bot-api-role-smpl', 'runtime': 'python3.7', 'memory': 128, 'timeout': 300, 's3_bucket': 'aws-syndicate-demo-bucket', 's3_key': 'yuliia_1703_1/telegram-bot-api-function-1.0.zip', 'env_vars': {'region': 'eu-west-1'}, 'vpc_sub_nets': None, 'vpc_security_group': None, 'dl_target_arn': None, 'tracing_mode': None, 'publish_version': False, 'layers': []} 2020-03-18 11:43:50,981 [DEBUG] sdct.syndicate.connection.helper,83 Traceback: Traceback (most recent call last): File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/helper.py", line 72, in wrapper return handler_func(args, kwargs) File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/lambda_connection.py", line 96, in create_lambda return self.client.create_function(params) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. 2020-03-18 11:43:52,308 [DEBUG] sdct.syndicate.connection.helper,78 Retry on create_lambda. Error: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. 2020-03-18 11:43:52,309 [DEBUG] sdct.syndicate.connection.helper,80 Parameters: (<syndicate.connection.lambda_connection.LambdaConnection object at 0x10aa79710>,), {'lambda_name': 'telegram-bot-api-function', 'func_name': 'handler.lambda_handler', 'role': 'arn:aws:iam::854412417832:role/sdct-telegram-bot-api-role-smpl', 'runtime': 'python3.7', 'memory': 128, 'timeout': 300, 's3_bucket': 'aws-syndicate-demo-bucket', 's3_key': 'yuliia_1703_1/telegram-bot-api-function-1.0.zip', 'env_vars': {'region': 'eu-west-1'}, 'vpc_sub_nets': None, 'vpc_security_group': None, 'dl_target_arn': None, 'tracing_mode': None, 'publish_version': False, 'layers': []} 2020-03-18 11:43:52,309 [DEBUG] sdct.syndicate.connection.helper,83 Traceback: Traceback (most recent call last): File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/helper.py", line 72, in wrapper return handler_func(args, kwargs) File "/Users/onsha/Work/epam/eoos_projects/aws-syndicate/syndicate/connection/lambda_connection.py", line 96, in create_lambda return self.client.create_function(params) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/onsha/Work/epam_v2/syndicate_rd/projects/syndicate-rd2-yuliia-hurova-moneybot/control-money-python-bot/.venv/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. 2020-03-18 11:43:56,670 [DEBUG] sdct.syndicate.core.resources.lambda_resource,249 Lambda created telegram-bot-api-function 2020-03-18 11:44:06,773 [DEBUG] sdct.syndicate.core.resources.lambda_resource,268 arn value: arn:aws:lambda:eu-west-1:854412417832:function:telegram-bot-api-function 2020-03-18 11:44:06,991 [INFO] sdct.syndicate.core.build.deployment_processor,294 AWS resources were deployed successfully 2020-03-18 11:44:06,992 [INFO] sdct.syndicate.core.build.deployment_processor,297 Going to apply dynamic changes 2020-03-18 11:44:06,992 [INFO] sdct.syndicate.core.build.deployment_processor,299 Dynamic changes were applied successfully 2020-03-18 11:44:06,992 [INFO] sdct.syndicate.core.build.deployment_processor,301 Going to create deploy output 2020-03-18 11:44:07,610 [INFO] sdct.syndicate.core.build.bundle_processor,62 Output file with name yuliia_1703_1/outputs/sdct.json has been created 2020-03-18 11:44:07,610 [INFO] sdct.syndicate.core.build.deployment_processor,307 Deploy output for sdct was created. 2020-03-18 11:44:07,610 [INFO] sdct.syndicate.core.helper,214 Stage deploy, elapsed time: 0:00:22.185713

The lambda was created after 2 retries - such time IAM and Lambda services need to sync latest data between each other.

oleksandr-onsha commented 4 years ago

Fix is available since v 0.8.5