Bug: parser does not returned a JSON.parsed() body payload for each record but the type is correct (SQS)

[cbarlow1993]

Expected Behavior

When using an envelope or an extended schema as per the example (see SQS schema in docs), then the event.Records[0].body should already be JSON.parsed() like the type suggests.

Current Behavior

The type is an object as per the schema, however it is still a string when using it, and then throws a type error on JSON.parse(record)

Code snippet

import { SqsRecordSchema, SqsSchema } from '@aws-lambda-powertools/parser/schemas';
import { JSONStringified } from '@aws-lambda-powertools/parser/helpers';

const messageBodySchema = z.object({
  property1: z.number(),
  property2: z.string(),
  property3: z.string().optional(),
});

const extendedSqsSchema = SqsSchema.extend({
  Records: z.array(
    SqsRecordSchema.extend({
      body: JSONStringified(messageBodySchema),
    }),
  ),
});

type ExtendedType = z.infer<typeof extendedSqsSchema>;

export const handler = async (event: ExtendedType, _context: Context): Promise<SQSBatchResponse> => {
  const batchItemFailures: SQSBatchItemFailure[] = [];
  for (const record of event.Records) {
    try {

      const { property1, property2, property3 } = record.body;

      console.log('property1', property1); //return undefined
      console.log('property2', property2); //return undefined
      console.log('property3', property3); //return undefined
....
})

export const exportedHandler = middy(handler).use(
  parser({ schema: extendedSqsSchema }),
);

Steps to Reproduce

1. make similar handler to above using v2.8.0

Possible Solution

I went through the code but it all seemed to make sense.

Powertools for AWS Lambda (TypeScript) version

latest

AWS Lambda function runtime

20.x

Packaging format used

npm

Execution logs

No response

[dreamorosi]

Hi @cbarlow1993, thanks for opening the issue.

I have tried to reproduce the behavior you described and I am unable to reproduce it.

I have created a sample project and used your code as function, although I had to make a couple of slight modifications to make it work:

+ import { z } from 'zod';
+ import type { Context, SQSBatchResponse, SQSBatchItemFailure } from 'aws-lambda';
+ import middy from '@middy/core';
+ import { parser } from '@aws-lambda-powertools/parser/middleware';
import { SqsRecordSchema, SqsSchema } from '@aws-lambda-powertools/parser/schemas';
import { JSONStringified } from '@aws-lambda-powertools/parser/helpers';

const messageBodySchema = z.object({
  property1: z.number(),
  property2: z.string(),
  property3: z.string().optional(),
});

const extendedSqsSchema = SqsSchema.extend({
  Records: z.array(
    SqsRecordSchema.extend({
      body: JSONStringified(messageBodySchema),
    }),
  ),
});

type ExtendedType = z.infer<typeof extendedSqsSchema>;

export const handler = async (event: ExtendedType, _context: Context): Promise<SQSBatchResponse> => {
  const batchItemFailures: SQSBatchItemFailure[] = [];
  for (const record of event.Records) {
    try {

      const { property1, property2, property3 } = record.body;

      console.log('property1', property1); //return undefined
      console.log('property2', property2); //return undefined
      console.log('property3', property3); //return undefined
- ....
- })
+    } finally {}
+  }
+
+  return { batchItemFailures };
+}

export const exportedHandler = middy(handler).use(
  parser({ schema: extendedSqsSchema }),
);

And then called the function directly using this payload:

{
  "Records": [
    {
      "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
      "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
      "body": "{\"property1\": 1, \"property2\": \"foo2\"}",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1545082649183",
        "SenderId": "AIDAIENQZJOLO23YVJ4VO",
        "ApproximateFirstReceiveTimestamp": "1545082649185"
      },
      "messageAttributes": {
        "testAttr": {
          "stringValue": "100",
          "binaryValue": "base64Str",
          "dataType": "Number"
        }
      },
      "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
      "awsRegion": "us-east-2"
    },
    {
      "messageId": "2e1424d4-f796-459a-8184-9c92662be6da",
      "receiptHandle": "AQEBzWwaftRI0KuVm4tP+/7q1rGgNqicHq...",
      "body": "{\"property1\": 1, \"property2\": \"foo2\", \"property3\": \"foo3\"}",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1545082650636",
        "SenderId": "AIDAIENQZJOLO23YVJ4VO",
        "ApproximateFirstReceiveTimestamp": "1545082650649",
        "DeadLetterQueueSourceArn": "arn:aws:sqs:us-east-2:123456789012:my-queue-dead"
      },
      "messageAttributes": {},
      "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
      "awsRegion": "us-east-2"
    }
  ]
}

which generates the following logs:

START RequestId: 61f48343-8e79-4b21-b47c-b930cb18aa6b Version: $LATEST
2024-09-17T13:07:37.205Z        61f48343-8e79-4b21-b47c-b930cb18aa6b    INFO    property1 1
2024-09-17T13:07:37.205Z        61f48343-8e79-4b21-b47c-b930cb18aa6b    INFO    property2 foo2
2024-09-17T13:07:37.206Z        61f48343-8e79-4b21-b47c-b930cb18aa6b    INFO    property3 undefined
2024-09-17T13:07:37.225Z        61f48343-8e79-4b21-b47c-b930cb18aa6b    INFO    property1 1
2024-09-17T13:07:37.225Z        61f48343-8e79-4b21-b47c-b930cb18aa6b    INFO    property2 foo2
2024-09-17T13:07:37.225Z        61f48343-8e79-4b21-b47c-b930cb18aa6b    INFO    property3 foo3
END RequestId: 61f48343-8e79-4b21-b47c-b930cb18aa6b
REPORT RequestId: 61f48343-8e79-4b21-b47c-b930cb18aa6b  Duration: 258.47 ms     Billed Duration: 259 ms Memory Size: 128 MB     Max Memory Used: 68 MB
XRAY TraceId: 1-66e97f19-3b3e8e1862442c2132468326       SegmentId: 13213422d11a9bbe     Sampled: true

Note that the first occurrence of property3 is undefined because the field is optional and wasn't included in the first message body.

After that, I also added an SQS queue as trigger for the function and put two messages into the queue using the AWS CLI:

aws sqs send-message \
  --queue-url https://sqs.eu-west-1.amazonaws.com/123456789012/MyQueue \
  --message-body '{"property1":1,"property2":"foo2"}'

and

aws sqs send-message \
  --queue-url https://sqs.eu-west-1.amazonaws.com/123456789012/MyQueue \
  --message-body '{"property1":1,"property2":"foo2","property3":"foo3"}'

and got the same outputs from the function.

I uploaded the sample that you can find here and deploy, if you want.

---

Based on the behavior you're describing, it appears that you might be calling the handler directly rather than the exportedHandler. Please confirm that your function is using that has handler in the function config.

Also, tangentially related, based on your use case you might want to check our Batch Processing utility which can help you process messages from SQS (among other sources) and handle partial failures more easily.

[cbarlow1993]

Hi @dreamorosi ,

You got it spot on. I can confirm the handler export was incorrect. And now I feel super stupid and apologies for wasting your time. Many thanks.

I've been using the utilities quite a bit and they've been excellent. In previous projects, we made our own Idempotency and Zod validator, but not as quite well designed as yours so we're making the switch.

I'll pay back the help with some suggested improvements to docs and more examples to how we've ended up using it after the project.

[github-actions[bot]]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[dreamorosi]

Hey @cbarlow1993, glad to hear you were able to find the issue! No time wasted at all, happy to help!

Thank you for your words, that's really great to hear! And yes, absolutely, if you spot any areas of improvements we're always open to improve.

And regarding the examples, it would also be really useful to learn how you use it.