Document SQS custom access policy

OkkarMin commented 3 years ago

Maintainer edit: a full solution is documented on this page here.

Hello,

Bottom Line Up Front:

Is it possible to pass a custom access policy to SQS queue created by lift's queue construct
Subscribe SQS queue created by lift's queue construct to SNS topic and apply filter policy for the subscription

Context: I am currently developing a system where

publisher -> SNS -> filter policy -> SQS -> lambda handler

I am using lift's queue construct to create SQS queue and its handler SQS -> lambda handler

service: email-service

provider:
  name: aws
  runtime: nodejs12.x
  region: ap-southeast-1
  stage: dev
  lambdaHashingVersion: 20201221

constructs:
  EmailQueue:
    type: queue
    worker:
      handler: emailQueueWorker.handler

plugins:
  - serverless-plugin-typescript
  - serverless-lift

Currently, I have been using AWS Console (web) to manually apply filter policy from SNS -> SQS as well as subscribe SQS to an SNS topic.

Then noticed that SNS was unable to send messages to SQS due to SQS created by lift queue construct not having a proper access control policy. I had to manually edit the access policy like below

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "Allow-SNS-SendMessage",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:*",
      "Resource": "arn:*****:email-service-dev-KeyShoeEmailQueue"
    }
  ]
}

Would there be a way to:

Pass custom access policy to SQS queue created by lift's queue construct? So that SNS can send messages to SQS
Subscribe SQS queue created by lift's construct to SNS topic with filter policy?

Thank you in advance!

mnapoli commented 3 years ago

Thanks for the details, that makes sense!

From what I understand, SQS queue policies are resources separate from queues (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-policy.html).

Is it possible to create a brand new queue policy in resources and make it apply to the Lift queue? For example (not tested!):

constructs:
  emailQueue:
    type: queue
    worker:
      handler: emailQueueWorker.handler

resources:
  Resources:
    ExtraQueuePolicy:
      Type: AWS::SQS::QueuePolicy
      Properties:
        Queues: [ ${construct:emailQueue.queueArn} ]
        PolicyDocument:
          ...

OkkarMin commented 3 years ago

Thanks for the example, let me test it out and get back to you!

OkkarMin commented 3 years ago

Hey! I was able to configure my use-case with the following serverless.yml config file 👍 (took me half a day to configure it properly)

Would it be useful to add a section on how to configure the SQS policy in the docs?

service: generic-service

provider:
  name: aws
  runtime: nodejs12.x
  region: ap-southeast-1
  stage: dev
  lambdaHashingVersion: 20201221

constructs:
  GenericQueue:
    type: queue
    worker:
      handler: genericQueueWorker.handler

resources:
  Resources:
    SubscribeToSNS:
      Type: AWS::SNS::Subscription
      Properties:
        Endpoint: ${construct:GenericQueue.queueArn}
        Protocol: sqs
        TopicArn: arn:of:sns:topic
        RawMessageDelivery: false
        FilterPolicy:
          EVENT_NAME:
            - "any"
            - "one"
            - "of"
            - "these"
    ExtraQueuePolicy:
      Type: AWS::SQS::QueuePolicy
      Properties:
        Queues:
          - ${construct:GenericQueue.queueUrl}
        PolicyDocument:
          Statement:
            - Sid: Allow-SNS-SendMessage-${self:service}
              Effect: Allow
              Principal:
                Service: sns.amazonaws.com
              Action:
                - sqs:*
              Resource: ${construct:GenericQueue.queueArn}

plugins:
  - serverless-plugin-typescript
  - serverless-lift

mnapoli commented 3 years ago

Thank you for sharing! And glad to hear it worked eventually.

I want to avoid adding too much to the documentation, here is what we can do: if more people have the same need, please add a comment to this issue.

If we discover more people need it (e.g. 2 more comments) we'll add it to the docs.

carvajalluis commented 2 years ago

attempting to apply this myself within the iam role in the provider but getting erros on calling ${construct...}

configValidationMode: error

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: "20201221"
  iam:
    role:
      statements:
        - Effect: Allow
          Principal:
            Service:
             - sns.amazonaws.com
          Action:
            - sqs:SendMessage
          Resource: ${construct:queue.queueArn} # <---HERE
          Condition:
            ArnEquals: !ImportValue sns-event-${sls:stage}

constructs:
  queue:
    type: queue
    worker:
      handler: handler.worker
    alarm: ${env:...}

plugins:
 ...
  - serverless-lift

resources:
  Resources:
    SnsSubscription:
      Type: AWS::SNS::Subscription
      Properties:
        Protocol: sqs
        Endpoint: ${construct:queue.queueArn}
        Region: ${aws:region}
        TopicArn: !ImportValue sns-event-${sls:stage}

I'm getting

 Serverless Error ----------------------------------------

  Configuration error at 'provider.iam.role.statements[0].Resource[0]': unsupported value

  Learn more about configuration validation here: http://slss.io/configuration-validation

  Get Support --------------------------------------------

my package json is

{
  "devDependencies": {
    "serverless": "^2.62.0",
    "serverless-lift": "^1.9.1",
    "serverless-python-requirements": "^5.1.1"
  }
}

OkkarMin commented 2 years ago

Hey @carvajalluis try to use ${construct:queue.queueUrl}

It worked for me when I used queueUrl instead of queueArn

And another way that helped me out was looking at the cloudformation generated by serverless in {serverless_app_folder}.serverless/cloudformation-template-update-stack.json to determine if the ${construct:queue.queueUrl} get successfully passed into the cloudformation

All the best and do let us know how it went for you!