Always getting ResourceConflictException creating functions

[antoniocaparros]

Type of request: This is a ...

[X] bug report

Detailed description

I am having problems deploying with serverless framework on my localstack. It is always getting this error:

localstack_main | 2021-04-21T06:02:41:DEBUG:localstack.utils.cloudformation.template_deployer: Error applying changes for CloudFormation stack "SmartHomeBackend-local": An error occurred (ResourceConflictException) when calling the CreateFunction operation: Function already exist: SmartHomeBackend-local-SmartHomeCore Traceback (most recent call last):
localstack_main |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1936, in _run
localstack_main |     self.do_apply_changes_in_loop(changes, stack, stack_name)
localstack_main |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1987, in do_apply_changes_in_loop
localstack_main |     self.apply_change(change, stack, new_resources, stack_name=stack_name)
localstack_main |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 2040, in apply_change
localstack_main |     result = deploy_resource(resource_id, new_resources, stack_name)
localstack_main |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1064, in deploy_resource
localstack_main |     return execute_resource_action(resource_id, resources, stack_name, ACTION_CREATE)
localstack_main |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1168, in execute_resource_action
localstack_main |     result = configure_resource_via_sdk(resource_id, resources, resource_type, func, stack_name, action_name)
localstack_main |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1289, in configure_resource_via_sdk
localstack_main |     raise e
localstack_main |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1284, in configure_resource_via_sdk
localstack_main |     result = function(**params)
localstack_main |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
localstack_main |     return self._make_api_call(operation_name, kwargs)
localstack_main |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 676, in _make_api_call
localstack_main |     raise error_class(parsed_response, operation_name)
localstack_main | botocore.errorfactory.ResourceConflictException: An error occurred (ResourceConflictException) when calling the CreateFunction operation: Function already exist: SmartHomeBackend-local-SmartHomeCore

I have checked before running this deploy that there aren't any resources created. My serverless.yml functions section:

SmartHomeCore:
  handler: src.main.core_handler
  timeout: 3
  memorySize: 128
  description: SmartHome Core - Manage every Core
  events:
    - http:
        path: /core/{proxy+}
        cors: true
        method: ANY
        private: false
        authorizer:
          type: COGNITO_USER_POOLS
          name: SmartHomeAuthorizer
          identitySource: method.request.header.X-API-Key
          arn: arn:aws:cognito-idp:us-east-1:${env:AWS_ACCOUNT_ID}:userpool/${env:COGNITO_POOL_ID}
    - http:
        path: /core/graphql
        cors: true
        method: ANY
        private: false
    - http:
        path: /core/ping
        method: GET
        private: false
    - http:
        path: /core/docs
        method: GET
        private: false
    - http:
        path: /core/redoc
        method: GET
        private: false
    - http:
        path: /core/openapi.json
        method: GET
        private: false
  environment:
    REGION: ${env:AWS_REGION}
    ACCOUNT_ID: ${env:AWS_ACCOUNT_ID}
    DATABASE_NAME: ${env:DATABASE_NAME}
    DATABASE_USER: ${env:DATABASE_USER}
    DATABASE_PORT: ${env:DATABASE_PORT}
    DATABASE_ENGINE: ${env:DATABASE_ENGINE}
    DATABASE_PASSWORD: ${env:DATABASE_PASSWORD}
    DATABASE_WRITER_HOST: ${env:DATABASE_WRITER_HOST}
    DATABASE_READER_HOST: ${env:DATABASE_READER_HOST}
    S3_MEDIA_URL: ${env:S3_MEDIA_URL}
    S3_MEDIA_FILES_BUCKET_NAME: ${env:S3_MEDIA_FILES_BUCKET_NAME}
  iamRoleStatementsName: SmartHomeCoreRole
  iamRoleStatements:
    - Effect: Allow
      Action:
        - logs:CreateLogGroup
        - logs:CreateLogStream
        - logs:PutLogEvents
      Resource: "*"
    - Effect: Allow
      Action:
        - s3:PutObject
      Resource: "*"
    - Effect: Allow
      Action:
        - iot:DeleteThing
        - iot:CreatePolicy
        - iot:DeletePolicyVersion
        - iot:CreatePolicyVersion
        - iot:ListPolicyVersions
        - iot:AttachPolicy
        - iot:DescribeThing
        - iot:RegisterCertificate
        - iot:ListThingPrincipals
        - iot:AttachThingPrincipal
        - iot:DetachThingPrincipal
        - iot:AddThingToThingGroup
        - iot:AttachPrincipalPolicy
        - iot:CreateKeysAndCertificate
      Resource: "*"
  layers:
    - { Ref: PythonRequirementsLambdaLayer }

My docker-compose.yml:

version: '2.1'

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack:latest
    network_mode: bridge
    ports:
      - "4566:4566"
      - "4571:4571"
      - "443:443"
      - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
    environment:
      - LOCALSTACK_SERVICES=acm,amplify,apigateway,apigatewayv2,application,appsync,athena,autoscaling,batch,cloudformation,cloudfront,cloudsearch,cloudtrail,cloudwatch,cloudwatchlogs,codecommit,cognito,cognito,docdb,dynamodb,dynamodbstreams,ec2,ecr,ecs,eks,elasticache,elasticbeanstalk,elb,elbv2,emr,es,events,firehose,glacier,glue,iam,iot,iotanalytics,iotevents,iot,iot,kafka,kinesis,kinesisanalytics,kms,lambda,logs,mediastore,neptune,organizations,qldb,rds,redshift,route53,s3,s3control,sagemaker,sagemaker,secretsmanager,ses,sns,sqs,ssm,stepfunctions,sts,timestream,transfer,xray
      - DEFAULT_REGION=eu-west-1
      - DEBUG=True
      - LOCALSTACK_API_KEY=${LOKALSTACK_APIKEY}
#      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
#      - DOCKER_HOST=unix:///var/run/docker.sock
#      - HOST_TMP_FOLDER=${TMPDIR}
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

Steps to reproduce

Run docker container TMPDIR=/private$TMPDIR docker-compose up. Deploy using serverless framework sls deploy --stage local --debug.

I am using plugin serverless-localstack for deployment

[whummer]

Thanks for reporting @antoniocaparros . We haven't been able to reproduce this issue so far. Can you please share a minimal reproducible example?

Also, please make sure to do a docker pull localstack/localstack to test with the latest version

Here's a repro that seems to be working:

$ touch handler.js
$ cat serverless.yml
service: lambda-test
frameworkVersion: "2"
plugins:
  - serverless-localstack
custom:
  localstack:
    stages:
      - local
package:
  include:
    - '**/**'
provider:
  name: aws
  region: us-east-1
  runtime: nodejs12.x
functions:
  SmartHomeCore:
    handler: handler.handler
    description: SmartHome Core - Manage every Core
    events:
      - http:
          path: /core/{proxy+}
          cors: true
          method: ANY
          private: false
          authorizer:
            type: COGNITO_USER_POOLS
            name: SmartHomeAuthorizer
            identitySource: method.request.header.X-API-Key
            arn: arn:aws:cognito-idp:us-east-1:000000000:userpool/pool_id
      - http:
          path: /core/graphql
          cors: true
          method: ANY
          private: false
      - http:
          path: /core/ping
          method: GET
          private: false
      - http:
          path: /core/docs
          method: GET
          private: false
    iamRoleStatementsName: SmartHomeCoreRole
    iamRoleStatements:
      - Effect: Allow
        Action:
          - logs:CreateLogGroup
          - logs:CreateLogStream
          - logs:PutLogEvents
        Resource: "*"
      - Effect: Allow
        Action:
          - s3:PutObject
        Resource: "*"
      - Effect: Allow
        Action:
          - iot:DeleteThing
          - iot:CreatePolicy
        Resource: "*"
    layers:
      - { Ref: MylayerLambdaLayer }
layers:
  mylayer:
    path: .
$ sls deploy --stage local
...
Serverless: Stack update finished...
...

Thanks!

[antoniocaparros]

Which extra info do You need?

[whummer]

@antoniocaparros Can you please share the version logs from your LocalStack container:

LocalStack version: 0.12.9.1
LocalStack Docker image id: 4b32cad49c25
LocalStack build date: 2021-04-22
LocalStack build git hash: b6203c02

Also, can you please share the full logs of your Docker container with - we may be able to reconstruct it from the DEBUG logs created by the CloudFormation deployment loop. Thanks for your help!

[paulrevere4]

Hey everyone, I see its been a while since the last activity on this issue, but I have some interesting findings that may be helpful. I ran into a similar issue when creating a lambda via the AWS SDK for Java 2.x though the issue would happen only occasionally.

I think it's related to retries on the AWS client. My hypothesis is that the localstack container is taking too long to respond to the first request to create the lambda and then the AWS client is retrying the request, localstack is then responding to the retry with the ResourceConflictException since it did actually create the lambda in the first request.

The AWS clients in the Java SDK seem to do 3 retries by default if nothing is configured based on these docs linked below (it sounds like it will use the "LEGACY" retry mode).

https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/retry/RetryMode.html

I was able to consistently recreate the issue by using a lambda client with the default retries and setting a really short timeout which will cause the client to retry the request to create the lambda.

LambdaClient client =
        LambdaClient.builder()
            // ... other overrides left out
            .overrideConfiguration(
                ClientOverrideConfiguration.builder()
                    .apiCallAttemptTimeout(
                        Duration.ofMillis(100)) // Too short for localstack to respond in time
                    .build())
            .build();

This consistently produces the ResourceConflictException when you call createFunction to make lambdas with this client.

software.amazon.awssdk.services.lambda.model.ResourceConflictException: Function already exist: FUNCTION_NAME_HERE (Service: Lambda, Status Code: 409, Request ID: null, Extended Request ID: null)

From that explanation it then follows that disabling the retries on the client should fix the issue.

LambdaClient client =
        LambdaClient.builder()
            // ... other overrides left out
            .overrideConfiguration(
                ClientOverrideConfiguration.builder()
                    .retryPolicy(RetryPolicy.builder().numRetries(0).build())
                    .build())
            .build();

I've tried this issue in my project and haven't seen the issue come back. Hopefully that helps your issue.