I've been looking at some of our error logs when using this library and noticed a good number of what just appears to be maybe flakiness in the aws kinesis library or just unhandled networking errors.
err.Error() == "shard shardId-000000000822 error: get records error: send request failed"
As a short-term solution, we ended up just wrapping the consumer initialization in a for loop and restarting the consumer when it gets these occasional errors (~50x per week for one of our services), but I was thinking it would be nice to potentially handle these errors at the shard level so that the whole consumer doesn't need to be cancelled/restarted.
Also related, a couple months ago, AWS had the kinesis outage in the US-EAST-1 region and we experienced roughly a 50% error rate with any requests to kinesis. If we were able to handle these errors at the shard level without impacting other shards, it would have given our service a chance to cope with the 50% error rate and still process the kinesis stream. Unfortunately, 50% error rate across several shards caused the consumer to stop and restart too frequently and we were stuck waiting until the error rate improved in Kinesis.
So I guess I have two questions:
Does this "send request failed" error look like something we can just add as an always retry-able error (I've seen that there is a small collection of those)?
Or, how receptive would you be to there being an optional error handler (isRetryable() method) that the caller can provide for handling the shard reader errors in a custom way? (I'd be happy to send a PR)
I've been looking at some of our error logs when using this library and noticed a good number of what just appears to be maybe flakiness in the aws kinesis library or just unhandled networking errors. err.Error() == "shard shardId-000000000822 error: get records error: send request failed"
As a short-term solution, we ended up just wrapping the consumer initialization in a for loop and restarting the consumer when it gets these occasional errors (~50x per week for one of our services), but I was thinking it would be nice to potentially handle these errors at the shard level so that the whole consumer doesn't need to be cancelled/restarted.
Also related, a couple months ago, AWS had the kinesis outage in the US-EAST-1 region and we experienced roughly a 50% error rate with any requests to kinesis. If we were able to handle these errors at the shard level without impacting other shards, it would have given our service a chance to cope with the 50% error rate and still process the kinesis stream. Unfortunately, 50% error rate across several shards caused the consumer to stop and restart too frequently and we were stuck waiting until the error rate improved in Kinesis.
So I guess I have two questions: