search

aws-samples / aws-lambda-fanout

A sample AWS Lambda function that accepts messages from an Amazon Kinesis Stream and transfers the messages to another data transport.
Apache License 2.0
288 stars 60 forks source link

Retry failed Kinesis putRecords #34

Closed dustinwilcock closed 6 years ago

dustinwilcock commented 6 years ago

post-kinesis.js now detects failed puts (e.g. ProvisionedThroughputExceededException) and retries with backoff (npm install retry) see Handling Failures When Using PutRecords in https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-sdk.html

(The other changes here, seem to be changes that were made inline to the Lambda, that hadn't yet been committed to this repo, including the SNS max size tweak. I started by cloning the repository locally, then downloading Lambda code into my local repo.)

ozbillwang commented 5 years ago

@dustinwilcock

What happen about this PR?

I currently have question about the retry strategy, such as:

If connection to one of fanout targets failed, what action should be followed?

should we record the sequence in some where, such as a dynamodb table, for each fanout target? so if lambda function is triggered, it can fanout the message start from next sequence specified, more than the latest message? So we can guarantee to sync the messages to destination always?

Or the lamdba restries and fanout the latest messages to all targets again until all targets get the messages?

dustinwilcock commented 5 years ago

I mistakenly submitted this PR to the origin repostitory. I meant to submit it on our own private fork. Thus why it's closed, and branch deleted. I apologize for any confusion.