I've tracked down more of the rate limiting issues, there was actually no problem in the logic, the rate limitation was on the DescribeLogStreams call, the API supports only 5 per second globally...
This call is necessary to get the current upload token to a log stream, the first implementation I had was making a call for each stream before calling PutLogEvents so it could figure out what upload token it needed to send...
This PR introduces a more complex logic to maintain tokens in cache and only fetch them for the first batch of message sent or when an error occurs.
I also noticed that PutLogEvents does return an error containing the upload token when none is provided so I added logic to parse the error message and retry in that case.
Please take a look and let me know if anything needs to be changed.
@segmentio/infra
I've tracked down more of the rate limiting issues, there was actually no problem in the logic, the rate limitation was on the DescribeLogStreams call, the API supports only 5 per second globally...
This call is necessary to get the current upload token to a log stream, the first implementation I had was making a call for each stream before calling PutLogEvents so it could figure out what upload token it needed to send...
This PR introduces a more complex logic to maintain tokens in cache and only fetch them for the first batch of message sent or when an error occurs. I also noticed that PutLogEvents does return an error containing the upload token when none is provided so I added logic to parse the error message and retry in that case.
Please take a look and let me know if anything needs to be changed.