search

awslabs / aws-fluent-plugin-kinesis

Amazon Kinesis output plugin for Fluentd
Apache License 2.0
293 stars 96 forks source link

Record size limit exceeded in 1883 KB #201

Closed Shravankumark closed 4 years ago

Shravankumark commented 4 years ago

Hi,

I am running into an issue with kinesis firehose plugin when there is large amount of log data. I know there is a limitation of 1MB on firehose side. Can you please help me in resolving the issue?

Config <match **> @type kinesis_firehose aws_key_id
aws_sec_key delivery_stream_name prod_firehose region us-east-1

flush_thread_count 2

Error: [error]: #0 Record size limit exceeded in 1883 KB:{"level":"INFO","message":"MMJ.misc.SQSQueue - PUT /event/v1/putnew Object exceeds the SQS message size limit:, evType:INFOEXCHANGE_UPDATED, activityType: INTEGRATION, subjectID:, posted: 2020-05-25T16:18:09.879Z, , invokedByMainEventQueueProcessor: false\n<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<infoExchange xmlns:ns2=\"urn:MMJ:sit\">\n \n MapInToDeliver\n \n \n \n CANONICAL\n \n true\n \n 2020-05-25T16:17:57.783Z\n

simukappu commented 4 years ago

Closing this issue for now. Please reopen if required.

singhajit89 commented 3 years ago

No solution/work around found for this issue yet.

simukappu commented 3 years ago

Discussing in #210.