CreateStack as true is deploying to other region(s) without the deployment package uploaded to corresponding region's S3 bucket

[malli1246]

I tried to setup a multi region serverless app which has a Lambda function that serves api requests and uses DynamoDB.

While deploying thru CI/CD tool for one of the region,

1. Global Table scripts started execution after the completion of stack updates for one of the desired regions.
2. Next, script started creating/updating stack of other region without uploading the artifacts/deployment package to next region.
3. The stack creation failed saying that the S3Key(path to package) as NoSuckKey and the Lambda deployment is failing causing the stack to rollback.

Please find the screenshots below for the stack updates verbose and the aws CFN error event from my sample project. Here, I tried to deploy to the default region but has two regions under custom node.

Didn't face any issue if there is no lambda to be created in the stack as the Lambda creation has a constraint that the deployment package should be in the same region.

Due to my delivery deadlines, had to search if there is a way and found another package for creating global table which just creates a table and create a replication for the same.

Any updates or any suggestions related to this issue is much appreciated

serverless.yml:

service: my-auth-service-globaltable-issue

plugins:
      - serverless-create-global-dynamodb-table

custom:
  globalTables:
    regions: # list of regions in which you want to set up global tables
      - us-east-1
      - us-west-2
    createStack: true
    tags: # List of tags that needs to applied to the new table (optional)
      - Key: tag-key
        Value: tag-value
      - Key: tag-key-2
        Value: tag-value-2

provider:
  name: aws
  runtime: python3.8

  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:DescribeStream
        - dynamodb:GetRecords
        - dynamodb:GetShardIterator
        - dynamodb:ListStreams
      Resource: arn:aws:dynamodb:*:*:table/usersTable-final
      # you can overwrite defaults here
  stage: dev
  region: us-east-1

functions:
  hello-final:
    handler: handler.hello

resources: # CloudFormation template syntax
  Resources:
    usersTable:
      Type: AWS::DynamoDB::Table      
      Properties:
        TableName: usersTable-globaltable-issue
        AttributeDefinitions:
          - AttributeName: email
            AttributeType: S
        KeySchema:
          - AttributeName: email
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 3
          WriteCapacityUnits: 3
        StreamSpecification:
          StreamViewType: NEW_AND_OLD_IMAGES

[rrahul963]

@malli1246 - seems like you are trying to create the dynamodb table and lambda function in the same serverless.yml file. When you try and setup global table, the plugin deploys the complete stack in the new replication region and in this case since your stack has lambda function definition as well, it tries to deploy that as well. Based on the error it seems like when cfn is trying to create lambda fn in new region its not able to access/find the deployment pkg.

Can you please check the following

1. is your s3 bucket accesable from the replication region
2. the lambda pkg name - does it contain region in the name

Alternatively, if you set createStack to false then plugin will use AWS SDK to create the tables in replication regions and not try and deploy lambda fn there.

[mtimbs]

I've also run into this issue. I don't think you can use the createStack: true flag because if I understand it correctly CloudFormation expects a seperate s3 deployment bucket for each region and so your stack won't be able to find anything in the second region as this plugin doesn't trigger uploading the artefacts to each region.

createStack: false also won't deploy the stacks but it will create the tables in all the regions. This means you can't really use this plugin on first deploy. Region 1 will deploy, create the table in all the regions and then when region 2 attempts to deploy the CloudFormation stack will error as the table already exists....

I have only been able to get this to work by doing individual region deploys and then pulling in this plugin. It's not the best DX and not really any easier than manually creating a global table.

Let me know if you'd like someone to look at this and I will give it a crack submitting a PR

[rrahul963]

@mtimbs - you are correct, when you do createStack: true, the plugin will take the whole cfn stack and try to deploy it in new region and with createStack: false, plugin will just create the dynamodb tables in new region and setup sync between the 2.

The plugin was built on an idea that app (lambda) stack is always separate from db (or any other infra) stack. This is also a good practice since your infra is not going to change as frequently as your app.

Let me know your thoughts.

[mtimbs]

when you do createStack: true, the plugin will take the whole cfn stack and try to deploy it in new region

Serverless framework makes assumptions around the naming of your s3 buckets, so unless you specify the bucket for your deployments this plugin will break (first) deploys and will need to be installed/included after the initial deployment. I think this is fine but maybe worth mentioning in the documentation? (Happy to PR that in if you agree)

The plugin was built on an idea that app (lambda) stack is always separate from db (or any other infra) stack. This is also a good practice since your infra is not going to change as frequently as your app.

I think this is a valid position to take, however it would be nice to be explicit about this (again happy to PR this in). Serverless is still very new and a lot of "best practices" are still evolving. In a traditional microservices/SOA architecture it is quite common to have a seperate table per service that is private to that service, so it is also valid to include Dynamo infrastructure with your application code since only 1-2 lambdas within that service might have read/write access to the table.

I feel like there's scope to evolve this plugin to be more generic, or to take the position that it does what it does and just be clear around the limitations and intended use cases.

[rrahul963]

@mtimbs - if you have a solution in mind then please fell free to open a pr.

i would prefer to keep the plugin focused on what the name suggests i.e. just focused on dynamodb replication but i do get your point about keeping infra with the api specially when developing smaller applications.

[rrahul963]

@mtimbs - i am closing this issue now, feel free to open a PR if you want to contribute back.