search

awslabs / amazon-kinesis-producer

Amazon Kinesis Producer Library
Apache License 2.0
402 stars 331 forks source link

Message Larger than 1 MB #185

Open ssabnis opened 6 years ago

ssabnis commented 6 years ago

I have a requirement to sent a JSON payload which is exceeding 1MB to Kinesis Stream, any suggestions how to achieve this scenario.

Thanks

cheefoo commented 6 years ago

One option will be to split the JSON payload into multiple parts and embed an Id in each part that can be used to identify these parts for coupling together once they are consumed

benalta commented 6 years ago

Hi, Bear in mind the 1MB is the record limit so if your event/record size is greater than 1MB, here are the options:

  1. Compress the json file prior on the producer side. (gzip ration can be ~x6)
  2. store the json file on s3 and send to kinesis a record with the json file metadata (bucket url) and the consumer can retrieve it within the consumer application logic/
  3. Split the file as was mentioned by @cheefoo
ssabnis commented 6 years ago

@benalta Thanks for the quick reply, can you emphasize on the point 2. is there an example you can share. I appreciate that.

Thank you.

VinothTS commented 5 years ago

@ssabnis : Were you able to achieve it ? If yes, what method have you followed

Asamkhata071 commented 5 years ago

I have a similar requirement, Is KPL a good option or should one use Kinesis Streams API with AWS-SDK ?. I know putRecords can send a row of maximum size 1MB but what is the maximum row size with KPL , is the row size with KPL fixed to 50 KB or is 1MB unless one uses aggregation ?.

aliartiza75 commented 4 years ago

I think in this scenario we should use Amazon MSK.