Enable Large Messages using S3

[adammead]

Reassembling log messages that have been split into multiple SQS messages can be problematic, so can we use S3 to store those large messages?

I see two options:

1. Use Amazon SQS Extended Client Library for Java to store the messages in S3 (either all or just the large ones) and then use the same library to read those messages (See https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html)
2. Write messages larger than 256 KB to S3 and do not send those messages on the SQS queue

I think both options are valid and could be supported so different methods of reading the messages can be supported. For instance, Logstash has an S3 input and an SQS input, but does not use the extended client library for Java, so it cannot read those references from the queue correctly.

[adesjardin]

We should support the following options related to large messages ( >256kb):

• Truncate and publish the message to the queue
• Discard the message and log a warning (Avoid truncating structured messages that won't be valid if truncated)
• FIFO Queue (Existing functionality)
• SQS Extended Client - Use the extended java client to publish the message via SQS+S3
• S3 Bucket - Publish each message to a defined S3 bucket as an individual object there to allow something else such as logstash to monitor the bucket and read in each message

[neckron]

Hi Guys I have one proposal for it:

1. add a parameter on SQS Appender to accept an option to decide which strategy apply over the incoming logger event.
   • add option parameter to hold strategy to follow.
   • remove largeMessageEnabled flag. I think it will not longer needed.
   • add parameters for AWS bucket location , name, etc pending

<Appenders>
    <SQS name="SQS" 
         awsAccessKey="<your AWS access key>" 
         awsRegion="<your AWS region>" 
         awsSecretKey="<your AWS secret key>" 
         awsS3Location="pending"
         maxBatchOpenMs="10000" 
         maxBatchSize="5" 
         maxInflightOutboundBatches="5" 
         queueName="<your AWS SQS queue name>"
         option="truncate|discard|fifo|extended|s3"
         largeMessageQueueName="<your AWS SQS FIFO queue name>"
         maxMessageBytes="250">
        <PatternLayout pattern="%-5p %d [%t] %c: ##MESSAGE## %m%n"/>
    </SQS>
</Appenders>

2. To support the options, the main idea is to create several utility classes which will contain the logic to transform the message accordingly.
3. in SQSAppender class it's needed a small refactor to select the utility class depending the incoming parameter. More details at image below.
4. A small refactor needs to be done on SQSManager to extract FIFO functionality to an utility class. SQSManager class should be responsible just for send messages to queue, truncating message logic should be done separately.
5. An AWSS3 utility class should be created to stablish a client conection to S3.
6. An utility class should be created to manage messages for SQS Extension.

leaving a desing here as image (hope it helps)

[neckron]

Hi, I was revisiting this, seems it's working well. Could I close it? @adesjardin @adammead