More than one s3 event failing deployment for existing s3 bucket.

[tarunjangra]

This is a Bug Report

Description

I have 30 different suffixes for which i want to trigger event associated with S3 and lambda. So i need to define all of them in serverless.yml file. It is working fine for one event but not working when i put two or more events. I think it is not working for multiple events.

• What went wrong? Throwing following error:

  
  Serverless Error ---------------------------------------
  
  An error occurred: GICreateThumbnailsCustomS31 - Failed to create resource. The statement id (GICreateThumbnails-gi3-staging-original) provided already exists. Please provide a new statement id, or remove the existing statement. See details in CloudWatch Log: 2019/07/27/[$LATEST]337e42c16b0643bcadd452556a6dc004.

- What did you expect should have happened?
I am expecting to see all events should be visible on my lambda console in AWS which i have defined in serverless.yml
- What was the config you used?
```yml
service: User
frameworkVersion: ">=1.48.4"
tenant: ginvoicing
app: ginvoicing

custom:
  bucket: gi3-original

provider:
  name: aws
  runtime: nodejs8.10
  memorySize: 192
  timeout: 10
    iamRoleStatements:
    - Effect: "Allow"
      Action:
        - s3:*
      Resource: "*"
  logRetentionInDays: 5
  stage: dev
  region: ap-south-1
functions:
  GICreateThumbnails:
    handler: GICreateThumbnails.handler
    name: GICreateThumbnails # optional, Deployed Lambda name
    description: To create thumbnail of the uploaded picture on s3. # optional, Description to publish to AWS
    events:
      - s3:
          bucket: ${self:custom.bucket}
          event: s3:ObjectCreated:*
          rules:
            - prefix: images/items/
            - suffix: .png
          existing: true   
      - s3:
          bucket: ${self:custom.bucket}
          event: s3:ObjectCreated:*
          rules:
            - prefix: images/user/
            - suffix: .png
          existing: true                

• What stacktrace or error message from your provider did you see?

  
  Stack Trace --------------------------------------------

ServerlessError: An error occurred: GICreateThumbnailsCustomS31 - Failed to create resource. The statement id (GICreateThumbnails-gi3-staging-original) provided already exists. Please provide a new statement id, or remove the existing statement. See details in CloudWatch Log: 2019/07/27/[$LATEST]337e42c16b0643bcadd452556a6dc004. at provider.request.then.data (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:121:33) From previous event: at AwsDeploy.monitorStack (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:27:12) at provider.request.then.cfData (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:103:28) From previous event: at AwsDeploy.update (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:103:8) From previous event: at AwsDeploy.BbPromise.bind.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:117:35) From previous event: at AwsDeploy.updateStack (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:113:33) From previous event: at AwsDeploy.BbPromise.bind.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:127:39) From previous event: at Object.aws:deploy:deploy:updateStack [as hook] (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:123:30) at BbPromise.reduce (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:464:55) From previous event: at PluginManager.invoke (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:464:22) at PluginManager.spawn (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:484:17) at AwsDeploy.BbPromise.bind.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:93:48) From previous event: at Object.deploy:deploy [as hook] (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:89:30) at BbPromise.reduce (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:464:55) From previous event: at PluginManager.invoke (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:464:22) at PluginManager.run (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:496:17) at variables.populateService.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/Serverless.js:116:33) at runCallback (timers.js:810:20) at tryOnImmediate (timers.js:768:5) at processImmediate [as _immediateCallback] (timers.js:745:5) From previous event: at Serverless.run (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/Serverless.js:103:74) at serverless.init.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/bin/serverless.js:52:28) at /Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:111:16 at /Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:45:10 at FSReqWrap.oncomplete (fs.js:135:15) From previous event: at initializeErrorReporter.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/bin/serverless.js:52:6) at runCallback (timers.js:810:20) at tryOnImmediate (timers.js:768:5) at processImmediate [as _immediateCallback] (timers.js:745:5) From previous event: at Object. (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/bin/serverless.js:38:39) at Module._compile (module.js:652:30) at Object.Module._extensions..js (module.js:663:10) at Module.load (module.js:565:32) at tryModuleLoad (module.js:505:12) at Function.Module._load (module.js:497:3) at Function.Module.runMain (module.js:693:10) at startup (bootstrap_node.js:191:16) at bootstrap_node.js:612:3

Similar or dependent issues:

- #6369

## Additional Data

- **_Serverless Framework Version you're using_**: 1.48.4
- **_Operating System_**: darwin
- **_Enterprise Plugin Versione_**: 1.3.3
- **_Platform SDK Version_**: 2.1.0

[michelem09]

I have same problem with existing S3, it seems to be random, when I deploy my app sometimes I receive CREATE_FAILED for existing S3 and the deploy fails. If I start it again with same environment it succeeds.

Your Environment Information

 Operating System:          darwin
 Node Version:              8.10.0
 Serverless Version:        1.48.4
 Enterprise Plugin Version: 1.3.2
 Platform SDK Version:      2.1.0

EDIT: To be clear I have multiple functions pointing to the same S3 bucket but with different suffixes.

[pmuens]

Thanks for opening @tarunjangra :+1:

~Cross-linking an answer I wrote in another, related issue here --> https://github.com/serverless/serverless/issues/6455#issuecomment-515957121~

~The TL;DR is that the stack got out of sync and the permission was already created by a prior attempt (while possibly still failing to setup everything correctly). I'll look into this to see if we can fix this somehow.~

Actually I was wrong. We're currently generating the statement id based on the function name and the bucket name. Therefore we end up with 2 identical statement ids. Let me see if I can provide a bugfix for that.

[pmuens]

@tarunjangra @michelem09 I worked on a fix today. Could you deploy with this branch to see if it resolves the problem on your end?

https://github.com/serverless/serverless/pull/6456

(You can install this via npm by running npm i -g serverless/serverless#existing-s3-fixes)

It did for me, but I want to make sure that it's also working with your config. We can merge it and release it with tomorrows release if everything is working on your end as well.

[michelem09]

Updating the dev deploy I'm still receiving CREATE_FAILED for existing S3 and I think I have to remove the triggers manually to fix this now.

Failed to create resource. The statement id (xxxx-dev-yyyy-bucketname) provided already exists. Please provide a new statement id, or remove the existing statement. See details in CloudWatch Log: 2019/07/29/[$LATEST]a514660d4b4d45d3xxxxxxxxxxxxxx

[pmuens]

Thanks for the update @michelem09 :+1:

Yes, I too see this error when I upgrade from an old deployment. Right now it's just working if you deploy from scratch. I try to mitigate this shortcoming so that you can upgrade without running into this issue.

[pmuens]

@tarunjangra @michelem09 this seems to be more involved than I thought...

I'll open up another PR tomorrow where I'll tacke this problem in isolation... I'll keep you posted.

[tarunjangra]

@pmuens Thank you to keep me updated about this issue.

For the time being, I have decided to go with new bucket instead of existing. It works perfectly for new bucket scenario. Although i am not sure why it is working for new bucket because first time bucket could be new but next time it is of course not new. But it worked.

Anyway, I tried to execute previous scenario with existing bucket as well by updating code as recommended by @pmuens. I am also getting "CREATE_FAILED" for existing S3 bucket.

[pmuens]

@tarunjangra @michelem09 I think I got it. Could you try the updated https://github.com/serverless/serverless/pull/6477 again? I wrote it in a way so that it should be backwards compatible. I'm not running into the CREATE errors again...

Thanks in advance!

[tarunjangra]

are you still working in the same branch. I tried npm i -g serverless/serverless#existing-s3-fixes to pull the latest code. but it is failing.

[michelem09]

@tarunjangra it was deployed on another branch npm i -g serverless/serverless#existing-s3-multi

[pmuens]

Shoot. Sorry for the confusion. Yes, I decided to split the 2 PRs up because they tackled different fixes.

[michelem09]

@pmuens unfortunately I received the CREATE_FAILED again at first try. Probably, as usual, at second try it will work. I'll update you.

[tarunjangra]

Here is my current configuration:

service: GIImageProcessing
frameworkVersion: ">=1.48.4"

custom:
  bucket:
    dev: gi-mu-staging
    prod: gi-mu-production

provider:
  name: aws
  runtime: nodejs8.10
  memorySize: 192
  timeout: 10
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - s3:*
      Resource: "*"
  logRetentionInDays: 1
  stage: ${opt:stage,'dev'}
  region: ap-south-1
functions:
  GICreateThumbnails:
    handler: src/GICreateThumbnails.handler
    name: GICreateThumbnails # optional, Deployed Lambda name
    description: To create thumbnail of the uploaded picture on s3. # optional, Description to publish to AWS
    events:
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectCreated:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .png
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectCreated:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpeg
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectCreated:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpg
  GIDeleteThumbnails:
    handler: src/GIDeleteThumbnails.handler
    name: GIDeleteThumbnails # optional, Deployed Lambda name
    description: To Delete thumbnail of the image on s3 bucket. # optional, Description to publish to AWS
    events:
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectRemoved:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .png
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectRemoved:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpeg
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectRemoved:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpg

Error i am getting

An error occurred: GICreateThumbnailsCustomS31 - Failed to create resource. Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type. See details in CloudWatch Log: 2019/07/31/[$LATEST]347e2745543a4efc8a563cef3f08cef2.

when I remove existing and run again. same configuration works perfectly.

[pmuens]

Thanks for trying that out @tarunjangra and @michelem09 :+1:

Damn. That's frustrating. It worked on my end (migrating from the old way of deploying the setup to the new way as reworked in the PR). Maybe my setup is too simplistic 🤔

@michelem09 are you able to share your serverless.yml as well (in a redacted form) so that I can try to reproduce the issue on my end. Thanks!

[HumbleBeck]

Hi, I'm getting similar error, but now with #6477 instead of conflict I'm getting An error occurred: OnVideoUploadedCustomS31 - Failed to update resource. Unable to validate the following destination configurations Maybe this info might help somehow.

[michelem09]

I have same error:

Failed to create resource. Unable to validate the following destination configurations

Here s my redacted config (I removed custom part and some functions):

# Welcome to Serverless!
#
# This file is the main config file for your service.
# It's very minimal at this point and uses default values.
# You can always add more config options for more control.
# We've included some commented out config examples here.
# Just uncomment any of them to get that config option.
#
# For full config options, check the docs:
#    docs.serverless.com
#
# Happy Coding!

service: publishTools

provider:
  name: aws
  runtime: nodejs8.10
  endpointType: edge
  vpc:
    securityGroupIds:
      - sg-5d3e0e24
    subnetIds:
      - subnet-95f079fd
      - subnet-9df079f5
      - subnet-8cf079e4
  stage: ${opt:stage, 'dev'}
  region: eu-west-1
  profile: default
  role: arn:aws:iam::123433347672:role/lambda_execution_s3_dynamo
  environment:
    DB_HOST: ${file(./config/${self:provider.stage}.json):dbHost}
    DB_USER: ${file(./config/${self:provider.stage}.json):dbUser}
    DB_PASS: ${file(./config/${self:provider.stage}.json):dbPass}
    DB_NAME: ${file(./config/${self:provider.stage}.json):dbName}
    DEBUG: ${file(./config/${self:provider.stage}.json):debug}
    SNS_ARN: ${file(./config/${self:provider.stage}.json):snsArn}
  apiKeys:
    - name: ${self:provider.stage}-publish-tools
      value: ${self:custom.apiKey}
  usagePlan:
    name: ${self:provider.stage}-publish-tools

  resourcePolicy:
    - Effect: Allow
      Principal: "*"
      Action: execute-api:Invoke
      Resource:
        - execute-api:/*/*/*
      Condition:
        StringNotEquals:
          aws:sourceVpc: "vpce-019730d2af5de95fc"

package:
  exclude:
    - layer/**
    - node_modules/**
    - backup/**
    - config/**
    - .gitignore
    - .git
    - bitbucket-pipelines.yml
    - package.json
    - package-lock.json
    - yarn.lock
    - README.md

functions:
  check:
    handler: api/check.handler
    memorySize: 128
    runtime: nodejs8.10
    timeout: 10
    layers:
      - { Ref: PublishToolsCommonLambdaLayer }
    events:
      - http: 
          path: check
          method: post
          private: true
          cors: true
          request:
            schema:
              application/json: ${file(./config/create_request.json)}
  checkAudio:
    handler: api/audio.handler
    memorySize: 3008
    runtime: nodejs8.10
    timeout: 180
    layers:
      - { Ref: PublishToolsCommonLambdaLayer }
    events:
      - sns: ${self:custom.snsArn}
      - s3:
          existing: true
          bucket: ${self:custom.audio.bucket}
          events:
            - s3:ObjectCreated:*
          rules:
            - prefix: ${self:custom.audio.prefix}
            - suffix: ${self:custom.audio.suffix}
      - http: 
          path: check/audio
          method: post
          private: true
          cors: true
          request:
            schema:
              application/json: ${file(./config/create_request.json)}

layers:
  publishToolsCommon:
    path: layer
    description: A Layer for node.js modules.
    compatibleRuntimes:
      - nodejs8.10
  javaPacks:
    path: java
    description: A Layer with java binaries.

plugins:
  - serverless-domain-manager

[pmuens]

Thanks for sharing your setup @michelem09 and thanks for confirming @HumbleBeck :+1:

@HumbleBeck can you provide your serverless.yml (or the critical parts of it)? I'm trying to derive a pattern here. I already have a hunch, but the more data, the better.

@michelem09 @tarunjangra is the following error new? Or have you seen it in the past?

Failed to create resource. Unable to validate the following destination configurations

[HumbleBeck]

Hi, this is partial serverless config, let me know if you need more info @pmuens

onImageUploaded:
  handler: functions/v1/listeners/imageUploadedListener.handler
  iamRoleStatements:
    - Effect: Allow
      Action:
        - lambda:GetLayerVersion
      Resource:
        - arn:aws:lambda:us-east-1::layer:sharp:3
  layers:
    - arn:aws:lambda:us-east-1::layer:sharp:3
  events:
    - s3:
        bucket: ${self:custom.BUCKET_NAME}
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: avatar/
          - suffix: original
    - s3:
        bucket: ${self:custom.BUCKET_NAME}
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: video/
          - suffix: thumbnail
    - s3:
        bucket: ${self:custom.BUCKET_NAME}
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: bio/
          - suffix: thumbnail
onVideoUploaded:
  handler: functions/v1/listeners/videoUploadedListener.handler
  environment:
    PIPELINE_ID: ${self:custom.VTPIPELINE_ID.${self:provider.stage}}
  iamRoleStatements:
    - Effect: Allow
      Action:
        - elastictranscoder:CreateJob
      Resource:
        - Fn::Join:
            - ""
            - - "arn:aws:elastictranscoder:"
              - Ref: AWS::Region
              - ":"
              - Ref: AWS::AccountId
              - ":pipeline/"
              - ${self:custom.VTPIPELINE_ID.${self:provider.stage}}

        - Fn::Join:
            - ""
            - - "arn:aws:elastictranscoder:"
              - Ref: AWS::Region
              - ":"
              - Ref: AWS::AccountId
              - ":preset/*"
  events:
    - s3:
        bucket: ${self:custom.BUCKET_NAME}
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: video/
          - suffix: original
    - s3:
        bucket: ${self:custom.BUCKET_NAME}
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: bio/
          - suffix: original

onVideoTranscoded:
  handler: functions/v1/listeners/videoTranscodedListener.handler
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:GetItem
      Resource:
        Fn::GetAtt:
          - OrderTable
          - Arn
  events:
    - s3:
        bucket: ${self:custom.BUCKET_NAME}
        existing: true
        event: s3:ObjectCreated:*
        rules:
          - prefix: video/
          - suffix: video

[michelem09]

@pmuens that error started since I used the branch version (existing-s3-fixes before and existing-s3-multi now).

Another interesting thing is this, after update is completed it tries to delete existing S3 and of course it fails but the update is completed then (GREEN).

CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - publishTools-dev
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::LayerVersion - JavaPacksLambdaLayer
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::LayerVersion - PublishToolsCommonLambdaLayer
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::LayerVersion - JavaPacksLambdaLayer
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::LayerVersion - JavaPacksLambdaLayer
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::LayerVersion - PublishToolsCommonLambdaLayer
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::LayerVersion - PublishToolsCommonLambdaLayer
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - CheckZipLambdaFunction
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - CheckLambdaFunction
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - CheckEpubLambdaFunction
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - CheckPdfLambdaFunction
CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - CheckAudioLambdaFunction
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - CheckZipLambdaFunction
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - CheckLambdaFunction
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - CheckPdfLambdaFunction
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - CheckAudioLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - CheckZipCustomS32
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - CheckPdfCustomS32
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - CheckAudioCustomS32
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - CheckPdfCustomS32
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - CheckZipCustomS32
CloudFormation - CREATE_COMPLETE - Custom::S3 - CheckPdfCustomS32
CloudFormation - CREATE_COMPLETE - Custom::S3 - CheckZipCustomS32
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - CheckAudioCustomS32
CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - CheckEpubLambdaFunction
CloudFormation - CREATE_COMPLETE - Custom::S3 - CheckAudioCustomS32
CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1564585102312
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - CheckEpubCustomS32
CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1564585102312
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Deployment - ApiGatewayDeployment1564585102312
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - CheckEpubCustomS32
CloudFormation - CREATE_COMPLETE - Custom::S3 - CheckEpubCustomS32
CloudFormation - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - publishTools-dev
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckZipCustomS3
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckAudioCustomS3
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1564410730594
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckPdfCustomS3
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::LayerVersion - JavaPacksLambdaLayer
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::LayerVersion - PublishToolsCommonLambdaLayer
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckEpubCustomS3
CloudFormation - DELETE_COMPLETE - AWS::ApiGateway::Deployment - ApiGatewayDeployment1564410730594
CloudFormation - DELETE_COMPLETE - AWS::Lambda::LayerVersion - JavaPacksLambdaLayer
CloudFormation - DELETE_COMPLETE - AWS::Lambda::LayerVersion - PublishToolsCommonLambdaLayer
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - CheckPdfCustomS3
CloudFormation - DELETE_COMPLETE - AWS::CloudFormation::CustomResource - CheckAudioCustomS3
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - CheckZipCustomS3
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - CheckEpubCustomS3
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckZipCustomS3
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckPdfCustomS3
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckEpubCustomS3
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - CheckEpubCustomS3
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - CheckPdfCustomS3
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - CheckZipCustomS3
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckZipCustomS3
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckPdfCustomS3
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - CheckEpubCustomS3
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - CheckEpubCustomS3
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - CheckPdfCustomS3
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - CheckZipCustomS3
CloudFormation - UPDATE_COMPLETE - AWS::CloudFormation::Stack - publishTools-dev
Serverless: Stack update finished...

[michelem09]

As side note the DELETE_FAILED errors looks like these:

Failed to delete resource. Unable to validate the following destination configurations See details in CloudWatch Log: 2019/07/31/[$LATEST]6dc735da74954a66a02c997454e3b573

And then (repeated two times):

Failed to delete resource. Statement publishTools-dev-checkEpub-bucket-name is not found in resource policy. See details in CloudWatch Log: 2019/07/31/[$LATEST]4ed11b8a59f348f98f932e53f38dd578

[pmuens]

@tarunjangra @michelem09 @HumbleBeck thanks for the in-depth error reporting.

I was finally able to reproduce the issue. It was caused by a race condition. I updated the PR which now includes the fix and the error seems to be resolved (I also added an integration test to cover this case). I know it might be annoying, but can one of you maybe test this PR again. TBH I'm not sure if there's a way to test this deterministically since we already deployed some many different versions in the past which failed one way or another and the stack might be in a weird state. Anyway, I would be really grateful if one can jump in and give it a shot. 🙏

Thanks again for taking the time to take a deep dive into this issue. It's certainly not that straightforward but I believe we're onto something here! 👍 💯

[HumbleBeck]

Thank you @pmuens :tada: However I've already redeployed bucket and event functions with old existing-s3-multi fixes, had to deploy once without them, then again with, and it went through. I just deployed with new existing-s3-multi fixes and it went through as well, but I cannot test it with old version anymore.

[tarunjangra]

@pmuens I am also in the same boat and do not have old version of the configuration. But I will create new configuration similar to the old one and try that fix. Expect my reply by tonight.

[pmuens]

Great! Thanks for confirming @HumbleBeck and thanks for reproducing this @tarunjangra :+1:

Hopefully this final fix is the last one we have to make in order to get this into a stable state!

[tcastelli]

Sorry to ping you @pmuens , but will this allow to add different buckets to the same function, or should that be another feature request?

[tarunjangra]

@pmuens I tested it thoroughly and got new issue. I am not sure if i should create new issue or should open the same one. Just to make you sure, I have installed master branch and it already has that fixed branch merged.

serverless deploy --verbose

Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service GIImageProcessing.zip file to S3 (935.89 KB)...
Serverless: Uploading custom CloudFormation resources...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - GIImageProcessing-dev
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - GICreateThumbnailsLambdaVersionhjRhQtuMscRr1F6MS9FvGWgqaRGyyzWrerjXScGhcGY
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - GICreateThumbnailsCustomS31
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - GICreateThumbnailsLambdaVersionhjRhQtuMscRr1F6MS9FvGWgqaRGyyzWrerjXScGhcGY
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - GICreateThumbnailsLambdaVersionhjRhQtuMscRr1F6MS9FvGWgqaRGyyzWrerjXScGhcGY
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - GIDeleteThumbnailsLambdaVersionEuE9vug8attALyPDc62yVYPXcnJKrbBc4a5kqGVlk
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - GIDeleteThumbnailsLambdaVersionEuE9vug8attALyPDc62yVYPXcnJKrbBc4a5kqGVlk
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - GIDeleteThumbnailsLambdaVersionEuE9vug8attALyPDc62yVYPXcnJKrbBc4a5kqGVlk
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - GICreateThumbnailsCustomS31
CloudFormation - CREATE_FAILED - Custom::S3 - GICreateThumbnailsCustomS31
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - GIImageProcessing-dev
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - GIImageProcessing-dev
CloudFormation - DELETE_SKIPPED - AWS::Lambda::Version - GIDeleteThumbnailsLambdaVersionEuE9vug8attALyPDc62yVYPXcnJKrbBc4a5kqGVlk
CloudFormation - DELETE_SKIPPED - AWS::Lambda::Version - GICreateThumbnailsLambdaVersionhjRhQtuMscRr1F6MS9FvGWgqaRGyyzWrerjXScGhcGY
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - GICreateThumbnailsCustomS31
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - GICreateThumbnailsCustomS31
CloudFormation - DELETE_IN_PROGRESS - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - DELETE_COMPLETE - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - DELETE_IN_PROGRESS - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - DELETE_IN_PROGRESS - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - DELETE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - DELETE_COMPLETE - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - DELETE_COMPLETE - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - DELETE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - GICreateThumbnailsCustomS31
CloudFormation - DELETE_COMPLETE - AWS::CloudFormation::CustomResource - GICreateThumbnailsCustomS31
CloudFormation - UPDATE_ROLLBACK_COMPLETE - AWS::CloudFormation::Stack - GIImageProcessing-dev

serverless.yml

service: GIImageProcessing
frameworkVersion: ">=1.48.4"

custom:
  enterprise:
    collectLambdaLogs: false
  bucket:
    dev: gi-mu-staging
    prod: gi-mu-production

provider:
  name: aws
  runtime: nodejs8.10
  memorySize: 192
  timeout: 10
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - s3:*
      Resource: "*"
  logRetentionInDays: 1
  stage: ${opt:stage,'dev'}
  region: ap-south-1
functions:
  GICreateThumbnails:
    handler: src/GICreateThumbnails.handler
    name: GICreateThumbnails # optional, Deployed Lambda name
    description: To create thumbnail of the uploaded picture on s3. # optional, Description to publish to AWS
    events:
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectCreated:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .png
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectCreated:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpeg
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectCreated:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpg
  GIDeleteThumbnails:
    handler: src/GIDeleteThumbnails.handler
    name: GIDeleteThumbnails # optional, Deployed Lambda name
    description: To Delete thumbnail of the image on s3 bucket. # optional, Description to publish to AWS
    events:
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectRemoved:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .png
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectRemoved:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpeg
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectRemoved:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpg

Error I am getting now.

Serverless Error ---------------------------------------

  An error occurred: GICreateThumbnailsCustomS31 - Failed to create resource. Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type. See details in CloudWatch Log: 2019/08/08/[$LATEST]1fe4381b53114851a8280926fe6ade69.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              8.11.2
     Framework Version:         1.49.0
     Plugin Version:            1.3.8
     SDK Version:               2.1.0

[tarunjangra]

I thought, i should mention that too. Without having "existing:true" same configuration works perfectly for the new s3 bucket.

[jack1902]

So from reading through this, I understand that this issue is present for existing buckets but not present for buckets that don't exist yet?

Ok so this is working if i reference the bucket name using the custom config, but if i use:

!Ref CustomBucket

To pull the name out, it runs into the object Object error

[ihorfito]

this is not fixed

[tcastelli]

yep still failing with same prefixes are added to the same bucket on different functions, would yo mind reopening or creating a new one @tarunjangra ?

[ihorfito]

solved: check s3 bucket: configuration -> events. Delete if you have some events there, sls deploy works as expected.

[dkvasnicka]

solved: check s3 bucket: configuration -> events. Delete if you have some events there, sls deploy works as expected.

Not everyone has the option to delete existing events. Some buckets need to have several different events hooked on them, for legacy reasons. I found out that it works if there are no overlaps in prefixes and suffixes.

[jorenvh1]

Removing the events worked for me but now it's changing all events where I only specified PUT,POST and MULTIPART to objectcreated:*, anyone knows how to fix this?

[c10h22]

We have another issue here when using Reference to an already created bucket:

events:
    - s3:
        bucket:
          Ref: PropertyRawDataBucket
        event: s3:ObjectCreated:*
        existing: true
        rules:
          - prefix: stayDays/
    - s3:
        bucket:
          Ref: PropertyRawDataBucket
        event: s3:ObjectCreated:*
        existing: true
        rules:
          - prefix: billing-items/

During deployment we get an error: Only one S3 Bucket can be configured per function. In "XXXXXXXXXX" you're attempting to configure "[object Object]" and "[object Object]" at the same time.

This is due to strict compare between the buckets here: https://github.com/serverless/serverless/blob/6619eb7cffe8f4fec8ca0f8d63a0d7af923a05be/lib/plugins/aws/package/compile/events/s3/index.js#L228-L234

[jorenvh1]

@c10h22 make sure you remove all your existing triggers you manually created before running sls deploy.

[c10h22]

@jorenvh1 already tested as suggested per your last comment. The root cause, is that S3 events plugin is comparing two objects bucket !== currentBucketName as it consider that we will always provide a string as bucket.

In my case, I am using {Ref: BucketLogicalId} which fails the previous check as object1 !== object2 will always return true which throws the error.

[brian-lesschwab]

@c10h22 I know it's not the answer we are looking for but pinning the serverless version to 1.57.0 seems to work for me

[anshul1790]

Hi @pmuens, @tarunjangra

I am having the same issue, in the first deployment, it goes really smooth as adding the invocation from s3 bucket.

In subsequent deployments, it gives me this error: "Failed to create resource. Configuration is ambiguously defined. It cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type. See details in CloudWatch Log"

I have filed the same issue here: https://forum.serverless.com/t/issue-in-creating-the-lambda-invocation-from-s3-bucket/10982

serverless.yml.zip

[syammohanp]

Same issue with Serverless 1.65.0 An error occurred: T*****S31 - Failed to create resource. Unable to validate the following destination configurations See details in CloudWatch Log: yyyy/mm/dd/[$LATEST]6b9b0cceb17e400fb8fb318b09af47b4

How can I see this cloudwatch log without selecting a log group?

Thanks and Regards Syam

[tommedema]

Also getting this, version 2.8.0

Where can we actually find these cloudwatch logs?

[emilianomongelo]

I'm also experiencing the same problem