search

awslabs / amazon-kinesis-agent

Continuously monitors a set of log files and sends new data to the Amazon Kinesis Stream and Amazon Kinesis Firehose in near-real-time.
Other
359 stars 223 forks source link

ProvisionedThroughputExceededExceptions are spammy! #41

Open diranged opened 8 years ago

diranged commented 8 years ago

See: https://github.com/awslabs/amazon-kinesis-agent/blob/6a88340b2ef4413538085738b3fa4ce0432bf394/src/com/amazon/kinesis/streaming/agent/tailing/KinesisSender.java#L121-L128

2016-07-13 21:32:40.435+0000 staging-us1-tools-xxx (sender-1100) com.amazon.kinesis.streaming.agent.tailing.KinesisSender [ERROR] kinesis:log-pipeline-eng-syslog:/mnt/log/*: Record returned error code ProvisionedThroughputExceededException: Rate exceeded for shard shardId-000000000001 in stream log-pipeline-eng-syslog under account xxx.

So ... when we're gleefully sending off logs to Kinesis and we suddenly bump into a temporary throttle, we are given tens of thousands of these log messages in just a matter of seconds:

[root@staging-us1-tools-xxx:/mnt/log:130]# grep ProvisionedThroughputExceededException /var/log/aws-kinesis-agent/aws-kinesis-agent.log  | wc
  13317  306291 5300166

I really feel like this code could be improved to write out a message like:

kinesis:log-pipeline-eng-syslog:/mnt/log/*: Record returned error code: ProvisionedThroughputExceededException (4924 times) ....

Otherwise this message is really spammy and has minimal usefulness. I'd like to see it reported once-per-patch, not once-per-message.

chaochenq commented 8 years ago

Yeah I agree that the messages could sometimes get crazy and annoying... I think it's a good idea to get the message published every 30 seconds. I've noted it down. Thanks!

chaochenq commented 8 years ago

I've pushed a commit that moves the per-record message to TRACE level. I think this can reduce the pain for now.

diranged commented 8 years ago

Thanks for that -- though it just hides it for now. I look forward to this being a "stat" message thats printed on-demand rather than once-per-event!

chaochenq commented 8 years ago

Yep I'll do that change later :)