Inspired by amazon-sqs-java-extended-client-lib
The AWS SQS XL Messages Library enables you to manage Amazon SQS message payloads with Amazon S3. This is especially useful for storing and retrieving messages with a message payload size greater than the current SQS limit of 256 KB, up to a maximum of 2 GB. Specifically, you can use this library to:
Specify whether message payloads are always stored in Amazon S3 or only when a message's size exceeds 256 KB.
Send a message that references a single message object stored in an Amazon S3 bucket.
Get the corresponding message object from an Amazon S3 bucket.
Delete the corresponding message object from an Amazon S3 bucket.
Add aws-sdk and aws-sqs-xl-messages
to your dependencies:
$ npm i aws-sdk aws-sqs-xl-messages
Decorate an SQS client from aws-sdk
with this library:
const { SQS, S3 } = require("aws-sdk"),
{ SQSExt, Config } = require("aws-sqs-xl-messages")(SQS), // inject the SQS client that will be decorated
config = new Config();
config.enableLargePayloadSupport(new S3(), "my-bucket"); // tell the client which S3 bucket to use.
// optionally tell the client whether it must always upload messages to S3. This defaults to false.
config.alwaysThroughS3 = false;
// optionally tell the client whether it must prefix S3 objects with the QueueName. Useful if you
// plan to use one single bucket for more than one SQS queue. This defaults to true.
config.addQueueToS3Key = true;
let sqs = new SQSExt({ extendedConfig: config });
// you can now use sqs as if it was an sqs client from aws-sdk
// will send message's body to S3 if it's larger than the threshold (or alwaysThroughS3)
await sqs.sendMessage(sendParams).promise();
// will fetch transparently message's body from S3 when needed.
const response = await sqs.receiveMessages(receiveParams).promise();
// will delete S3 object (if any) too.
await sqs.deleteMessage(deleteParams).promise();
When working with AWS Lambda, you don't need to call receiveMessage
and deleteMessage
manually. It's managed internally by AWS. That means we can't transparently download nor delete S3 objects. It needs to be done manually inside the lambda function.
To solve this issue, two methods are exposed: transformLambdaRecords
and deleteRecords
. They take care of downloading messages from S3; and removing them from S3 respectively.
const { SQS, S3 } = require("aws-sdk"),
{ SQSExt, Config } = require("aws-sqs-xl-messages")(SQS); // inject the SQS client that will be decorated
// lambda handler
const handler = async (event, context) => {
const config = new Config();
config.enableLargePayloadSupport(new S3(), "my-bucket"); // tell the client which S3 bucket to use.
const sqsClient = new SQSExt({ extendedConfig: config }),
records = await sqsClient.transformLambdaRecords(event.Records);
// Apply your lambda logic using records. message bodies are now the content from S3.
await sqsClient.deleteRecords(records);
};
module.exports = handler;
Optionally, an S3 lifecycle can be set up to delete S3 objects older than a threshold. So you don't need to call deleteRecords
directly.
v1.1.0 has a few limitations:
Only the following methods are decorated:
sendMessage
receiveMessage
deleteMessage
deleteMessageBatch
NOTE
We plan to extend this list at least with sendMessageBatch, purgeQueue and deleteQueue.
Decoration is done by overwriting AWS.Request methods, and some workflows are not yet supported. In particular, promise-like and in-place callbacks will work. However calling to send
manually won't.
// this works
await sqs.sendMessage(params).promise();
// this works
sqs.sendMessage(params, callback);
// this doesn't work
let request = sqs.sendMessage(params);
request.send(callback);