`extendedLibraryCompatibility` option is not compatible with AWS extended client library for Java

[onderceylan]

The same problem was reported earlier on issue https://github.com/aspecto-io/sns-sqs-big-payload/issues/19, but the fix does not make sns-sqs-big-payload compatible with the JSON schema used by https://github.com/awslabs/amazon-sqs-java-extended-client-lib.

What's the issue?

Amazon S3 reference in the message body of sns-sqs-big-payload does not match with the JSON schema of the Java library.

The JSON structure that Java client produces is in the following format;

[
  "software.amazon.payloadoffloading.PayloadS3Pointer",
  {
    "s3BucketName": "extended-client-bucket",
    "s3Key": "xxxx-xxxxx-xxxxx-xxxxxx"
  }
]

Whereas the format that sns-sqs-big-payload expects when extendedLibraryCompatibility is enabled is as follows;

{
  "s3BucketName": "extended-client-bucket",
  "s3Key": "xxxx-xxxxx-xxxxx-xxxxxx.json"
}

There are two notable differences;

1. Java library produces an array, but sns-sqs-big-payload expects an object.
2. sns-sqs-big-payload uses S3 key directly to fetch the object, and as the Java library's message doesn't have .json extension in the value of s3Key, it cannot resolve the object from S3.

How to reproduce?

Assuming there is an object in S3 with the name 7ac71e8a-35bd-420f-b644-97abf2317e1d.json, running the following script will help reproducing the errors in the library.

import {SQS} from 'aws-sdk';
const sqs = new SQS({region: sqsConfig.region});
await sqs
  .sendMessage({
    QueueUrl: sqsConfig.queueUrl,
    MessageBody:
      '["software.amazon.payloadoffloading.PayloadS3Pointer",{"s3BucketName":"extended-client-bucket","s3Key":"7ac71e8a-35bd-420f-b644-97abf2317e1d"}]',
    MessageAttributes: {
      SQSLargePayloadSize: {
        StringValue: '5198',
        StringListValues: [],
        BinaryListValues: [],
        DataType: 'Number',
      },
    },
  })
 .promise();

The first error you'd hit would be There were 2 validation errors:\n* MissingRequiredParameter: Missing required key 'Bucket' in params\n* MissingRequiredParameter: Missing required key 'Key' in params", because lib cannot map the Key and Bucket fields in the payload due to JSON schema mismatch.

And if you get over it with manual payload transformation, due to not having the .json extension in S3 key name, the lib will hit "NoSuchKey: The specified key does not exist. error.

What's the fix?

When the opt-in extendedLibraryCompatibility option is used, the library should respect the JSON schema which the AWS client library produces. And, it should add an additional postfix while fetching the S3 object.

[onderceylan]

Please let me know if you'd be interested in a PR that fixes those issues mentioned.

[YanivD]

@onderceylan Thanks for reporting this issue. We're interested in a PR to resolve those issues, contributions are very welcome!

[onderceylan]

@onderceylan Thanks for reporting this issue. We're interested in a PR to resolve those issues, contributions are very welcome!

Great, I'm going to submit a PR soon.

[onderceylan]

@YanivD can you please approve the PR https://github.com/aspecto-io/sns-sqs-big-payload/pull/43 for the workflow run, and also review the PR?