NerdWalletOSS / kinesis-python

Low level, multiprocessing based AWS Kinesis producer & consumer library
Other
118 stars 50 forks source link

Update MAX_COUNT to be the maximum number of allowed by PutRecords #13

Closed langseth closed 6 years ago

langseth commented 6 years ago

According to AWS documentation[1], when putting multiple records in a single call the maximum number of records are 500 for a single call. The previous count was based off of the max rate that kinesis shard can handle

[1] https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html

langseth commented 6 years ago

A bit more information here is the relevant documentation from the URL:

Each PutRecords request can support up to 500 records. Each record in the request can be as large as 1 MB, up to a limit of 5 MB for the entire request, including partition keys. Each shard can support writes up to 1,000 records per second, up to a maximum data write total of 1 MB per second.

-- https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html

Regardless of the message rate, which can be fixed by backing off and retrying, going over the 500 record limit for a single PutRecords call will always fail.

codecov-io commented 6 years ago

Codecov Report

Merging #13 into master will not change coverage. The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master      #13   +/-   ##
=======================================
  Coverage   34.02%   34.02%           
=======================================
  Files           4        4           
  Lines         288      288           
=======================================
  Hits           98       98           
  Misses        190      190
Impacted Files Coverage Δ
src/kinesis/producer.py 48.86% <100%> (ø) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 5305f7b...98cf893. Read the comment docs.

ibnbay00 commented 6 years ago

@borgstrom , Not merged yet?