amazon-archives / amazon-kinesis-connectors

Apache License 2.0
325 stars 191 forks source link

Check buffer after each record is added #75

Open jbeemster opened 7 years ago

jbeemster commented 7 years ago

When GetRecords is called it does not appear to adhere to the buffer limits that have been put in place. This results in Buffers which can in some cases far exceed what is safe for an emitter to actually send.

This results in either the need to re-split the buffer within the Emitter to ensure that the buffer size is within a manageable range or simply resulting in the emitter failing to push the Buffer to its target due to size constraints of the sink.