These changes are to address the log buffering fragility issue noted here.
The adjustment logic took a few different forms based on testing - i also have a version of this that keeps a queue of latest results if anyone is interested in going down that route let me know (I ended up trimming this down to a more simple mechanism after seeing the complexity there).
I am more than happy to take suggestions on the adjustment logic - it's easy to make the backoff more dramatic if desired but I found it to be safest to keep the changes to small increments, if the network is really in a bad state the batch limit will be adjusted eventually.
complications/lessons learned
increasing the batch limit:
We publish logs in batches every minute by log type. This means that while some log types may become backlogged others can always publish successfully - causing the batch limit to re-adjust upwards too quickly. To get around this we only increase the limit if the previous batch limit was both hit and published successfully.
decreasing the batch limit:
depending on how bad the network connection is and how it is terminated, any number of errors can pop out (not just network/transport errors, sometimes just failures to parse the response) . To get around this we keep track of when the batch was started, and only reduce the limit if an error occurred after a 20 second duration
These changes are to address the log buffering fragility issue noted here.
The adjustment logic took a few different forms based on testing - i also have a version of this that keeps a queue of latest results if anyone is interested in going down that route let me know (I ended up trimming this down to a more simple mechanism after seeing the complexity there).
I am more than happy to take suggestions on the adjustment logic - it's easy to make the backoff more dramatic if desired but I found it to be safest to keep the changes to small increments, if the network is really in a bad state the batch limit will be adjusted eventually.
complications/lessons learned