vmware-go-kcl is a vmware originated open-source project for AWS Kinesis Client Library in Go. It has been widely used by many external companies and internally by Carbon Black. vmware-go-kcl-v2 is its companion project by utilizing AWS Go SDK V2 which introduces lots of breaking changes. To keep the repo clean, it is better to have a separated repo vmware-go-kcl-v2 with better golang project structure improvement.
MIT License
23
stars
17
forks
source link
localTPSExceededError so sleep for a second spams the log output #40
Is your feature request related to a problem? Please describe.
This logging is happening way too many times per second to be Info level. I understand that this is going to happen once per thread, so that maybe a contributing factor here, but it is very intense in the logging. If you send your logs to somewhere like Sumologic or DataDog, this could be very expensive.
See attached gist with the log output snippit.
Describe the solution you'd like
I think this backoff time should be configurable, and one second may not even be enough for the default. If it is configurable, I can pass in 2 or 5 seconds, and this will actually alleviate the throttling issue it is hitting. Looking closer in the code, and at AWS limits, this is mitigating the 5 transactions per second limit, and makes sense to wait the amount of time it is.
A better option: Just make these logs debug level. If someone wants to see them, they can change the log level.
Describe alternatives you've considered
I could put a sleep time in my ProcessRecords implementation, but I don't know if that is good, bad, or indifferent...
Is your feature request related to a problem? Please describe.
This logging is happening way too many times per second to be Info level. I understand that this is going to happen once per thread, so that maybe a contributing factor here, but it is very intense in the logging. If you send your logs to somewhere like Sumologic or DataDog, this could be very expensive.
See attached gist with the log output snippit.
Describe the solution you'd like
I think this backoff time should be configurable, and one second may not even be enough for the default. If it is configurable, I can pass in 2 or 5 seconds, and this will actually alleviate the throttling issue it is hitting. Looking closer in the code, and at AWS limits, this is mitigating the 5 transactions per second limit, and makes sense to wait the amount of time it is.
A better option: Just make these logs debug level. If someone wants to see them, they can change the log level.
Describe alternatives you've considered
I could put a sleep time in my
ProcessRecordsimplementation, but I don't know if that is good, bad, or indifferent...Additional context
Log showing this happening many times a second. https://gist.github.com/komealy/039b6b5ff8be646aa5b68c3d164bf234#file-vmware-go-kcl-v2-log