Docs: "ReportBatchItemFailures" for Lambda SQS event sources

[brysontyrrell]

Describe your idea/feature/enhancement

The SQS event source for Lambda supports reporting failed messages within a batch: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting

Example response:

{ 
  "BatchItemFailures": [ 
        {
            "ItemIdentifier": "id2"
        },
        {
            "ItemIdentifier": "id4"
        }
    ]
}

The SQS event source created by SAM does not enable this option. This has great benefit when enabled so we can report back messages that failed without forcing the entire batch to be retried.

Here is an example event source created by SAM:

{
    "EventSourceMappings": [
        {
            "UUID": "8908d78f-2c2d-4d1d-b926-bf63cdb0d938",
            "BatchSize": 10,
            "MaximumBatchingWindowInSeconds": 10,
            "EventSourceArn": "arn:aws:sqs:us-east-2:XXXXXXXXXX:sqs-testing2-Queue-1G8UDVZB7G9X5",
            "FunctionArn": "arn:aws:lambda:us-east-2:XXXXXXXXXX:sqs:function:sqs-testing2-Poller-hYTxlVdWa5Lk",
            "LastModified": "2021-12-01T16:25:03.426000-08:00",
            "State": "Enabled",
            "StateTransitionReason": "USER_INITIATED",
            "FunctionResponseTypes": []
        }
]

Here is the same event source created manually and setting ReportBatchItemFailures:

{
    "EventSourceMappings": [
        {
            "UUID": "0880a914-4cf1-45d8-807a-58c8bf206262",
            "BatchSize": 10,
            "MaximumBatchingWindowInSeconds": 10,
            "EventSourceArn": "arn:aws:sqs:us-east-2:XXXXXXXXXX:sqs-testing-Queue-31LDB7P1N6O3",
            "FunctionArn": "arn:aws:lambda:us-east-2:XXXXXXXXXX:function:sqs-testing-Poller-gasD2e2vIiFQ",
            "LastModified": "2021-12-01T15:57:38.388000-08:00",
            "State": "Enabled",
            "StateTransitionReason": "USER_INITIATED",
            "FunctionResponseTypes": [
                "ReportBatchItemFailures"
            ]
        }
    ]
}

Proposal

Add an additional property onto SQS event configurations to allow enabling features like ReportBatchItemFailures.

Things to consider: [ ] The SAM documentation will need to be updated

[jfuss]

@brysontyrrell Thanks for the request but it doesn't look like something CloudFormation supports at this time. Until that is released, SAM can't do anything: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-permission.html

[brysontyrrell]

@jfuss It looks like it is in CloudFormation:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html#cfn-lambda-eventsourcemapping-functionresponsetypes

AWS::Lambda::EventSourceMapping

FunctionResponseTypes
(Streams and SQS) A list of current response type enums applied to the event source mapping.

Valid Values: ReportBatchItemFailures
Required: No
Type: List of String
Maximum: 1
Update requires: No interruption

[brysontyrrell]

Function response types has already been exposed for the DynamoDB event source in SAM:

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-dynamodb.html

[heitorlessa]

For anyone else looking at this for answers, here's a functional sample template that works (bonus points if you use Python with Lambda): https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/batch/#required-resources

[jfuss]

Sorry just getting back from parental leave.

@brysontyrrell I was looking at Permissions not Event Source Mappings 🤦

@heitorlessa and @brysontyrrell So then it looks like we support this already? Seems like the case quickly looking through code but our docs don't suggest it. I assume @heitorlessa tested that Powertools example. 🤔

[brysontyrrell]

@jfuss @heitorlessa - Hey guys. Any word on updating the documentation to reflect that this is in SAM and supported?

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html

This still doesn't show that FunctionResponseTypes is a supported attribute:

BatchSize: Integer
Enabled: Boolean
FilterCriteria: FilterCriteria
MaximumBatchingWindowInSeconds: Integer
Queue: String

[jfuss]

@brysontyrrell Looking at the commit: https://github.com/aws/serverless-application-model/pull/1883 it only mentions support for Kinesis and DynamoDB Streams. Maybe SQS came later?

Anyways, let me kick off an internal issue to get this added to SQS. Updating labels as well.

[jfalkenstein]

What's the status on this issue? This seems to still be an issue and the ticket's approaching it's 1 year mark...

[hoffa]

Reproducable example for posterity:

Transform: AWS::Serverless-2016-10-31
Resources:
  MyQueue:
    Type: AWS::SQS::Queue
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      InlineCode: foo
      Handler: bar
      Runtime: python3.9
      Events:
        SQSEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt MyQueue.Arn
            FunctionResponseTypes:
              - ReportBatchItemFailures

Deploy with:

sam deploy --region us-west-2 --resolve-s3 --capabilities CAPABILITY_IAM --stack-name test-2515 --template template.yaml

[hoffa]

It's there now: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html#sam-function-sqs-functionresponsetypes