Open adammead opened 2 years ago
We should support the following options related to large messages ( >256kb):
Hi Guys I have one proposal for it:
SQS
Appender to accept an option to decide which strategy apply over the incoming logger event.
option
parameter to hold strategy to follow.largeMessageEnabled
flag. I think it will not longer needed.<Appenders>
<SQS name="SQS"
awsAccessKey="<your AWS access key>"
awsRegion="<your AWS region>"
awsSecretKey="<your AWS secret key>"
awsS3Location="pending"
maxBatchOpenMs="10000"
maxBatchSize="5"
maxInflightOutboundBatches="5"
queueName="<your AWS SQS queue name>"
option="truncate|discard|fifo|extended|s3"
largeMessageQueueName="<your AWS SQS FIFO queue name>"
maxMessageBytes="250">
<PatternLayout pattern="%-5p %d [%t] %c: ##MESSAGE## %m%n"/>
</SQS>
</Appenders>
SQSAppender
class it's needed a small refactor to select the utility class depending the incoming parameter. More details at image below.SQSManager
to extract FIFO functionality to an utility class. SQSManager
class should be responsible just for send messages to queue, truncating message logic should be done separately.leaving a desing here as image (hope it helps)
Hi, I was revisiting this, seems it's working well. Could I close it? @adesjardin @adammead
Reassembling log messages that have been split into multiple SQS messages can be problematic, so can we use S3 to store those large messages?
I see two options:
I think both options are valid and could be supported so different methods of reading the messages can be supported. For instance, Logstash has an S3 input and an SQS input, but does not use the extended client library for Java, so it cannot read those references from the queue correctly.