We have an accesslogs package which handles the logistics of sending and storing log entries generated by access log middleware. Watched buckets and log destinations are configured by way of a gateway config setting.
Currently it works by creating "shipments", which accumulate individual log entry lines until the buffer reaches the limit (default: 64MiB). The way logs are shipped to the destination bucket is either if the buffer reaches the limit, or the gateway is shut down. This limit setting can be tuned so that log entries are flushed out sooner, but there is a point when very infrequent use means the buffer stays around for a very long time. This becomes more apparent when you load balance requests across multiple multiple gateways.
One idea is to introduce a time limit the shipment can be buffered for, after which it gets flushed.
Acceptance Criteria
If I configure bucket logging for a bucket and make only a few requests to it, these entries are shipped to the destination once the configured time interval has passed, even if the shipment limit has not been reached yet.
Existing shipment limit size still works as before.
Goal
We have an
accesslogspackage which handles the logistics of sending and storing log entries generated by access log middleware. Watched buckets and log destinations are configured by way of a gateway config setting.Currently it works by creating "shipments", which accumulate individual log entry lines until the buffer reaches the limit (default: 64MiB). The way logs are shipped to the destination bucket is either if the buffer reaches the limit, or the gateway is shut down. This limit setting can be tuned so that log entries are flushed out sooner, but there is a point when very infrequent use means the buffer stays around for a very long time. This becomes more apparent when you load balance requests across multiple multiple gateways.
One idea is to introduce a time limit the shipment can be buffered for, after which it gets flushed.
Acceptance Criteria