search

localstack / localstack

💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline
https://localstack.cloud
Other
53.79k stars 3.9k forks source link

Deploy fails due to ResourceConflictException/Function already exist #2470

Closed maxgr0 closed 3 years ago

maxgr0 commented 4 years ago

Type of request: This is a ...

[x] bug report [ ] feature request

Detailed description

My deploy to localstack running in Docker (deploy is made via serverless-localstack) fails every time. With version 0.11.1 it worked after 3-5 attempts (without changing anything) and with the latest version there is no change to get the whole stack deployed to localstack. My localstack docker image is the latest (last pulled 06/01/20 1pm) Error log from localstack:

localstack_1        | 2020-05-28T14:17:13:DEBUG:localstack.services.cloudformation.cloudformation_starter: Currently processing stack resource api-dev-dev/GraphQLMainHandlerLambdaFunction: None
localstack_1        | 2020-05-28T14:17:13:WARNING:localstack.services.awslambda.lambda_api: Function not found: arn:aws:lambda:eu-central-1:000000000000:function:api-dev-dev-graphQLMainHandler
localstack_1        | 2020-05-28T14:17:13:DEBUG:localstack.services.cloudformation.cloudformation_starter: Deploying CloudFormation resource (update=False, exists=False, updateable=False): {'Type': 'AWS::Lambda::Function', 'Properties': {'Code': {'S3Bucket': 'testbucket', 'S3Key': 'serverless/api-dev/dev/1590675423991-2020-05-28T14:17:03.991Z/api-dev.zip'}, 'FunctionName': 'api-dev-dev-graphQLMainHandler', 'Handler': 'src/handler/graphQLMainHandler.graphQLMainHandler', 'MemorySize': 1024, 'Role': 'arn:aws:iam::123456789012:role/api-dev-dev-eu-central-1-lambdaRole', 'Runtime': 'nodejs12.x', 'Timeout': 6, 'Environment': {'Variables': {'STAGE': 'dev', 'MAIN_TABLE_NAME': 'main-dev', 'MAIN_BUCKET_NAME': 'main-dev', 'AWS_REGION': 'eu-central-1'}}}, 'DependsOn': ['GraphQLMainHandlerLogGroup', 'IamRoleLambdaExecution']}
localstack_1        | 2020-05-28T14:17:13:DEBUG:localstack.utils.cloudformation.template_deployer: Running action "create" for resource type "Lambda::Function" id "GraphQLMainHandlerLambdaFunction"
localstack_1        | 2020-05-28T14:17:13:DEBUG:localstack.utils.cloudformation.template_deployer: Request for resource type "Lambda::Function" in region eu-central-1: create_function {'FunctionName': 'api-dev-dev-graphQLMainHandler', 'Runtime': 'nodejs12.x', 'Role': 'arn:aws:iam::000000000000:role/api-dev-dev-eu-central-1-lambdaRole', 'Handler': 'src/handler/graphQLMainHandler.graphQLMainHandler', 'Code': {'S3Bucket': 'testbucket', 'S3Key': 'serverless/api-dev/dev/1590675423991-2020-05-28T14:17:03.991Z/api-dev.zip'}, 'Environment': {'Variables': {'STAGE': 'dev', 'MAIN_TABLE_NAME': 'main-dev', 'MAIN_BUCKET_NAME': 'main-dev', 'AWS_REGION': 'eu-central-1'}}, 'Timeout': 6, 'MemorySize': 1024}
localstack_1        | 2020-05-28T14:17:13:DEBUG:localstack.services.awslambda.lambda_executors: Getting status for container "localstack_lambda_arn_aws_lambda_eu-central-1_000000000000_function_api-dev-dev-graphQLMainHandler": docker ps -a --filter name='localstack_lambda_arn_aws_lambda_eu-central-1_000000000000_function_api-dev-dev-graphQLMainHandler' --format "{{ .Status }} - {{ .Names }}" | grep -w "localstack_lambda_arn_aws_lambda_eu-central-1_000000000000_function_api-dev-dev-graphQLMainHandler" | cat
localstack_1        | 2020-05-28T14:17:24:INFO:localstack.services.awslambda.lambda_executors: Checking if there are idle containers.
localstack_1        | 2020-05-28T14:18:14:WARNING:localstack.services.awslambda.lambda_api: Function already exist: api-dev-dev-graphQLMainHandler
localstack_1        | 2020-05-28T14:18:14:WARNING:localstack.utils.cloudformation.template_deployer: Error calling <bound method ClientCreator._create_api_method.<locals>._api_call of <botocore.client.Lambda object at 0x7fe3f31c0f10>> with params: {'FunctionName': 'api-dev-dev-graphQLMainHandler', 'Runtime': 'nodejs12.x', 'Role': 'arn:aws:iam::000000000000:role/api-dev-dev-eu-central-1-lambdaRole', 'Handler': 'src/handler/graphQLMainHandler.graphQLMainHandler', 'Code': {'S3Bucket': 'testbucket', 'S3Key': 'serverless/api-dev/dev/1590675423991-2020-05-28T14:17:03.991Z/api-dev.zip'}, 'Environment': {'Variables': {'STAGE': 'dev', 'MAIN_TABLE_NAME': 'main-dev', 'MAIN_BUCKET_NAME': 'main-dev', 'AWS_REGION': 'eu-central-1'}}, 'Timeout': 6, 'MemorySize': 1024} for resource: {'Type': 'AWS::Lambda::Function', 'Properties': {'Code': {'S3Bucket': 'testbucket', 'S3Key': 'serverless/api-dev/dev/1590675423991-2020-05-28T14:17:03.991Z/api-dev.zip'}, 'FunctionName': 'api-dev-dev-graphQLMainHandler', 'Handler': 'src/handler/graphQLMainHandler.graphQLMainHandler', 'MemorySize': 1024, 'Role': 'arn:aws:iam::123456789012:role/api-dev-dev-eu-central-1-lambdaRole', 'Runtime': 'nodejs12.x', 'Timeout': 6, 'Environment': {'Variables': {'STAGE': 'dev', 'MAIN_TABLE_NAME': 'main-dev', 'MAIN_BUCKET_NAME': 'main-dev', 'AWS_REGION': 'eu-central-1'}}}, 'DependsOn': ['GraphQLMainHandlerLogGroup', 'IamRoleLambdaExecution']}
localstack_1        | 2020-05-28T14:18:14:ERROR:localstack.services.cloudformation.cloudformation_starter: Unable to parse and create resource "GraphQLMainHandlerLambdaFunction": An error occurred (ResourceConflictException) when calling the CreateFunction operation: Function already exist: api-dev-dev-graphQLMainHandler Traceback (most recent call last):
localstack_1        |   File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 202, in parse_and_create_resource
localstack_1        |     return _parse_and_create_resource(
localstack_1        |   File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 324, in _parse_and_create_resource
localstack_1        |     result = deploy_func(logical_id, resource_map_new, stack_name=stack_name)
localstack_1        |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1066, in deploy_resource
localstack_1        |     return execute_resource_action(resource_id, resources, stack_name, ACTION_CREATE)
localstack_1        |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1096, in execute_resource_action
localstack_1        |     result = configure_resource_via_sdk(resource_id, resources, resource_type, func, stack_name)
localstack_1        |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1173, in configure_resource_via_sdk
localstack_1        |     raise e
localstack_1        |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1170, in configure_resource_via_sdk
localstack_1        |     result = function(**params)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
localstack_1        |     return self._make_api_call(operation_name, kwargs)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 635, in _make_api_call
localstack_1        |     raise error_class(parsed_response, operation_name)
localstack_1        | botocore.errorfactory.ResourceConflictException: An error occurred (ResourceConflictException) when calling the CreateFunction operation: Function already exist: api-dev-dev-graphQLMainHandler
localstack_1        | 
localstack_1        | 2020-05-28T14:18:14:DEBUG:localstack.services.cloudformation.cloudformation_listener: Error response for CloudFormation action "CreateStack" (400) POST /: b'Function already exist: api-dev-dev-graphQLMainHandler'

serverless.yaml:

service:
  name: api-${self:provider.stage}

provider:
  name: aws
  runtime: nodejs12.x
  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'eu-central-1'}
  profile: dev-local
  environment:
    STAGE: ${self:provider.stage}
    MAIN_TABLE_NAME: ${self:custom.mainTableName}
    MAIN_BUCKET_NAME: ${self:custom.mainBucketName}
    #MAIN_TABLE_STREAM_ARN:
      #Fn::GetAtt:
        #- MainTable
        #- StreamArn
    #DYNAMODB_REGION: ${self:provider.region}
    #AWS_PROFILE: ${self:provider.profile}
    AWS_REGION: ${self:provider.region}

package:
  include:
    - src/**

functions:
 graphQLMainHandler:
   handler: src/handler/graphQLMainHandler.graphQLMainHandler
   events:
     - http:
         method: post
         path: graphql-main
         cors: true
 dynamoDBStreamHandler:
   handler: src/handler/dynamoDBStreamHandler.dynamoDBStreamHandler
   events:
     - stream:
         type: dynamodb
         arn:
           Fn::GetAtt:
             - MainTable
             - StreamArn
         batchSize: 10
         startingPosition: TRIM_HORIZON
 pictureHandler:
   handler: src/handler/pictureHandler.pictureHandler
   events:
     - http:
         method: get
         path: picture
         cors: true
 taskHandler:
   handler: src/handler/taskHandler.taskHandler
 updateMainTableElasticsearchMappings:
   handler: src/handler/dev.updateMainTableElasticsearchMappings
 createMainTableElasticsearchIndex:
   handler: src/handler/dev.createMainTableElasticsearchIndex
 analysisTaskRunner:
   handler: src/handler/taskRunner.analysisTaskRunner
 sendNotificationsTaskRunner:
   handler: src/handler/taskRunner.sendNotificationsTaskRunner

plugins:
  - serverless-webpack
  - serverless-localstack
  - serverless-offline
resources:
  Resources:
    MainTable:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: ${self:custom.mainTableName}
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        BillingMode: PAY_PER_REQUEST
        StreamSpecification:
          StreamViewType: NEW_IMAGE
    MainBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.mainBucketName}
        AccessControl: PublicRead
        CorsConfiguration:
          CorsRules:
            - AllowedMethods:
                - GET
              AllowedOrigins:
                - "*"
              AllowedHeaders:
                - "*"

custom:
  serverless-offline:
    host: 0.0.0.0
    port: 3000
  webpack:
    packager: yarn
    webpackConfig: ./webpack.config.js
    includeModules: true
  localstack:
    debug: true
    stages:
      - dev
    autostart: false
    #lambda:
      #mountCode: false

  mainTableName: main-${self:provider.stage}
  mainBucketName: main-${self:provider.stage}

Actual behavior

The deployment fails every time. I already found the issues #1694 #1677 #326 but could not figure out the problem either. The deployment directly to AWS works without any problems.

Command used to start LocalStack

TMPDIR=/private$TMPDIR docker-compose -f docker-compose.dev.yml up docker-compose.dev.yml:

version: '3'
services:
  localstack:
    privileged: true
    image: localstack/localstack
    environment:
      - EDGE_PORT=4566
      - DEFAULT_REGION=eu-central-1
      - AWS_DEFAULT_REGION=eu-central-1
      - DEBUG=1
      - COMPOSE_PARALLEL_LIMIT=100
      - LAMBDA_EXECUTOR=docker-reuse
      - DOCKER_HOST=unix:///var/run/docker.sock
      - USE_SSL=1
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
    ports:
      - "4566-4599:4566-4599"

(...)

Thanks in advance for any approaches.

┆Issue is synchronized with this Jira Bug by Unito

maxgr0 commented 4 years ago

Short update: deploying one or two functions alone works when the rest is commented out. It works even if I change the handler of all functions to a simple handler like in some serverless examples. So with an increasing size of the package, the error seems to come up. Digging into the source tonight, maybe I can find something. My first guess was that there are some timeouts while deploying because the complete package size is relatively big for eight functions and the deployment takes quite long. Another observation: When the error comes, there are 1-2 functions and the resources/streams already deployed but not accessible. Tried to give docker some more resources but that did not help either.

jasonstrimpel commented 4 years ago

I'm also experiencing this issue.

I've only started to dig into this today, but my hypothesis is that the fact that the account numbers in the ARNs for the Lambda and the IAM may have something to do with it:

arn:aws:lambda:eu-central-1:000000000000:function:api-dev-dev-graphQLMainHandler
arn:aws:iam::123456789012:role/api-dev-dev-eu-central-1-lambdaRole

I don't know how to change this or if it matters. Just an observation. It's interesting to note that the lambda does exist and can be invoked (at least in my case):

awslocal lambda list-functions
awslocal lambda invoke --function-name  function-name --payload '{}' /tmp/lambda.out
maxgr0 commented 4 years ago

@jasonstrimpel Invoking is no problem for the lambdas which are created. When trying to deploy these eight functions, one or two as well as the mentioned resources get created (and they are fully working). When having a simple handler like this and setting it for all functions as the handler, everything gets deployed without any problems:

import 'source-map-support/register';

export const hello: APIGatewayProxyHandler = async (event, _context) => {
  return {
    statusCode: 200,
    body: JSON.stringify({
      message: 'Go Serverless Webpack (Typescript) v1.0! Your function executed successfully!',
      input: event,
    }, null, 2),
  };
}

Switching then three functions (randomly) to the much more complex handlers (increasing the zip from 400kb to 40mb) works too. Switching the handler of a fourth function starts to throw errors and makes the whole stack not deployable. Thats a curiosity I do not really understand as deploying to AWS works without any problems too. Tried to activate the pro version of localstack (just to be sure 🤣) but that did not fix the problem either.

maxgr0 commented 4 years ago

Another observation: When first deploying with a simple handler, the handler is set to src/handler/testHandler.hello. After changing 2-3 handlers to the real handlers and re-deploying, the deploy works but the handler is not updated when using aws lambda list-functions --endpoint http://localhost:4566. After the deployment of these changed handlers (deploy is successful), there are coming up some errors from localstack:

Unable to parse and update resource "DynamoDBStreamHandlerLambdaFunction": Read timeout on endpoint URL: "https://c97113e0f113:4574/2015-03-31/functions/api-dev-dev-dynamoDBStreamHandler/code" Traceback (most recent call last):
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
localstack_1        |     six.raise_from(e, None)
localstack_1        |   File "<string>", line 3, in raise_from
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
localstack_1        |     httplib_response = conn.getresponse()
localstack_1        |   File "/usr/lib/python3.8/http/client.py", line 1322, in getresponse
localstack_1        |     response.begin()
localstack_1        |   File "/usr/lib/python3.8/http/client.py", line 303, in begin
localstack_1        |     version, status, reason = self._read_status()
localstack_1        |   File "/usr/lib/python3.8/http/client.py", line 264, in _read_status
localstack_1        |     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
localstack_1        |   File "/usr/lib/python3.8/socket.py", line 669, in readinto
localstack_1        |     return self._sock.recv_into(b)
localstack_1        |   File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
localstack_1        |     return self.read(nbytes, buffer)
localstack_1        |   File "/usr/lib/python3.8/ssl.py", line 1099, in read
localstack_1        |     return self._sslobj.read(len, buffer)
localstack_1        | socket.timeout: The read operation timed out
localstack_1        | 
localstack_1        | During handling of the above exception, another exception occurred:
localstack_1        | 
localstack_1        | Traceback (most recent call last):
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/httpsession.py", line 254, in send
localstack_1        |     urllib_response = conn.urlopen(
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
localstack_1        |     retries = retries.increment(
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 379, in increment
localstack_1        |     raise six.reraise(type(error), error, _stacktrace)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise
localstack_1        |     raise value
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
localstack_1        |     httplib_response = self._make_request(
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 428, in _make_request
localstack_1        |     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 335, in _raise_timeout
localstack_1        |     raise ReadTimeoutError(
localstack_1        | urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='c97113e0f113', port=4574): Read timed out. (read timeout=60)
localstack_1        | 
localstack_1        | During handling of the above exception, another exception occurred:
localstack_1        | 
localstack_1        | Traceback (most recent call last):
localstack_1        |   File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 212, in parse_and_update_resource
localstack_1        |     return _parse_and_create_resource(logical_id, resource_json, resources_map, region_name, update=True)
localstack_1        |   File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 324, in _parse_and_create_resource
localstack_1        |     result = deploy_func(logical_id, resource_map_new, stack_name=stack_name)
localstack_1        |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 994, in update_resource
localstack_1        |     client.update_function_code(FunctionName=props['FunctionName'], **props['Code'])
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
localstack_1        |     return self._make_api_call(operation_name, kwargs)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 621, in _make_api_call
localstack_1        |     http, parsed_response = self._make_request(
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 641, in _make_request
localstack_1        |     return self._endpoint.make_request(operation_model, request_dict)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/endpoint.py", line 102, in make_request
localstack_1        |     return self._send_request(request_dict, operation_model)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/endpoint.py", line 136, in _send_request
localstack_1        |     while self._needs_retry(attempts, operation_model, request_dict,
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/endpoint.py", line 253, in _needs_retry
localstack_1        |     responses = self._event_emitter.emit(
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/hooks.py", line 356, in emit
localstack_1        |     return self._emitter.emit(aliased_event_name, **kwargs)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
localstack_1        |     return self._emit(event_name, kwargs)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
localstack_1        |     response = handler(**kwargs)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 183, in __call__
localstack_1        |     if self._checker(attempts, response, caught_exception):
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 250, in __call__
localstack_1        |     should_retry = self._should_retry(attempt_number, response,
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 277, in _should_retry
localstack_1        |     return self._checker(attempt_number, response, caught_exception)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 316, in __call__
localstack_1        |     checker_response = checker(attempt_number, response,
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 222, in __call__
localstack_1        |     return self._check_caught_exception(
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
localstack_1        |     raise caught_exception
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/endpoint.py", line 200, in _do_get_response
localstack_1        |     http_response = self._send(request)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/endpoint.py", line 269, in _send
localstack_1        |     return self.http_session.send(request)
localstack_1        |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/httpsession.py", line 289, in send
localstack_1        |     raise ReadTimeoutError(endpoint_url=request.url, error=e)
localstack_1        | botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://c97113e0f113:4574/2015-03-31/functions/api-dev-dev-dynamoDBStreamHandler/code"

Iam running out of ideas, maybe @whummer has an approach on how to debug this case further🤓

zhenwenc commented 4 years ago

I am also experiencing similar error like this, when deploying lambda with multiple handlers, but failed on lambdaRole creation:

The issue was actually caused by a boolean value TRUE in the provider.environment section of the project serverless.yml:

stack_1    | 2020-06-12T21:50:45:ERROR:localstack.services.cloudformation.cloudformation_starter: Unable to parse and create resource "GraphqlLambdaFunction": Parameter validation failed:
stack_1    | Invalid type for parameter Environment.Variables.XXXXXX, value: True, type: <class 'bool'>, valid types: <class 'str'> Traceback (most recent call last):
stack_1    |   File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 202, in parse_and_create_resource
stack_1    |     return _parse_and_create_resource(
stack_1    |   File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 324, in _parse_and_create_resource
stack_1    |     result = deploy_func(logical_id, resource_map_new, stack_name=stack_name)
stack_1    |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1066, in deploy_resource
stack_1    |     return execute_resource_action(resource_id, resources, stack_name, ACTION_CREATE)
stack_1    |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1096, in execute_resource_action
stack_1    |     result = configure_resource_via_sdk(resource_id, resources, resource_type, func, stack_name)
stack_1    |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1173, in configure_resource_via_sdk
stack_1    |     raise e
stack_1    |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1170, in configure_resource_via_sdk
stack_1    |     result = function(**params)
stack_1    |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
stack_1    |     return self._make_api_call(operation_name, kwargs)
stack_1    |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 607, in _make_api_call
stack_1    |     request_dict = self._convert_to_request_dict(
stack_1    |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 655, in _convert_to_request_dict
stack_1    |     request_dict = self._serializer.serialize_to_request(
stack_1    |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/validate.py", line 297, in serialize_to_request
stack_1    |     raise ParamValidationError(report=report.generate_report())
stack_1    | botocore.exceptions.ParamValidationError: Parameter validation failed:
stack_1    | Invalid type for parameter Environment.Variables.XXXXXX, value: True, type: <class 'bool'>, valid types: <class 'str'>
whummer commented 4 years ago

Thanks for the detailed report @maxgr0 . Trying to figure out what could be the root cause for this issue. If I understand correctly, it seems to be likely a performance issue - as the problems start to surface if many functions are deployed in parallel, or if large Lambda ZIP files are being used.

Do you think you could prepare a small Github repo with a full, isolated sample that can be easily cloned and deployed via "sls deploy"? That would help us greatly in reproducing the issue. Thanks

Also - I noticed that you have mountCode: false commented out in your serverless config - were you able to give that a try, and what is the behavior if Lambda mounting is enabled?

maxgr0 commented 4 years ago

Thanks for the detailed report @maxgr0 . Trying to figure out what could be the root cause for this issue. If I understand correctly, it seems to be likely a performance issue - as the problems start to surface if many functions are deployed in parallel, or if large Lambda ZIP files are being used.

Do you think you could prepare a small Github repo with a full, isolated sample that can be easily cloned and deployed via "sls deploy"? That would help us greatly in reproducing the issue. Thanks

Also - I noticed that you have mountCode: false commented out in your serverless config - were you able to give that a try, and what is the behavior if Lambda mounting is enabled?

Hi @whummer! I've commented out mountCode: false because of some weird handler/resource not found errors while executing lambdas. Already thought of providing a repo to give more details but as the code is standard typescript and can't be public, Iam not able to do it. The problem does not occur when deploying many lambdas, just when the bundles are starting to get bigger. The weird thing about the whole thing is, that, after deploying around 10 times and getting an error, a deploy works and everything runs perfectly.

whummer commented 4 years ago

Thanks for the follow-up @maxgr0 . I understand that posting private/confidential code to a public repo is not an option - but would the problem be reproducible with a simple project that uses the same typescript setup and dependency stack as your project, but contains only "hello world" lambda handlers, instead of the actual handler code...? Thanks

sebarys commented 3 years ago

Hi @whummer ,

We're experiencing similar issue. In each test we're creating AWS lambda function where name is concatenation: test_$functionName where functionName = ${UUID.randomUUID().toString} - each created function is differ only by functionName, other params are the same for all initial created functions.

Our test very frequently failing because of 

software.amazon.awssdk.services.lambda.model.ResourceConflictException: Function already exist: FUNCTION_NAME_HERE (Service: Lambda, Status Code: 409, Request ID: null, Extended Request ID: null)

we're logging each invocation of function create and there is no duplicate create invocation for the same functionName so it is almost impossible that function exist before.

macnev2013 commented 3 years ago

Hello @sebarys, Thanks for reporting. Is it possible to provide a small GitHub repo with full, isolated reproducible example with deployment instructions ? It would help us a lot in reproducing the issue. Thanks.

whummer commented 3 years ago

@sebarys Closing, as this issue cannot be reproduced. Please report here with a smal sample repo (incl. Lambda handler files) to help us replicate this issue. Thanks!

paulrevere4 commented 2 years ago

I ran into an issue similar to @sebarys's issue and I think its related to retries the AWS client does, I explained in a comment on this issue: https://github.com/localstack/localstack/issues/3915#issuecomment-984091000.