We ran into some trouble when you upgrade from the existing DockerLogInput plugin to the current one on dev. That's because switching from using attach to logs on the Docker API means that we're going to pull in all the past logs for containers that in some cases have been up for quite some time. This leads to some pretty massive duplication of logs and a burying of the downstream logging system for awhile. Secondly, there is currently no way to expire entries from the sinces JSON file and so for systems with a lot of deployments, the list just grows and grows.
This PR fixes both of those issues by introducing some settings:
new_containers_replay_logs - Allows disabling of replay of logs for existing containers. This should be set to true when upgrading. It can later be switched back to help guarantee log delivery while Heka is not listening. Defaults to true
container_expiry_days - Represents the number of days to keep a container ID in the file after we last see it. Defaults to 30 days.
There is some additional refactoring of the code for encapsulation and readability. Finally it adds documentation for a missing setting introduced earlier, fields_from_labels.
We ran into some trouble when you upgrade from the existing
DockerLogInput
plugin to the current one ondev
. That's because switching from usingattach
tologs
on the Docker API means that we're going to pull in all the past logs for containers that in some cases have been up for quite some time. This leads to some pretty massive duplication of logs and a burying of the downstream logging system for awhile. Secondly, there is currently no way to expire entries from the sinces JSON file and so for systems with a lot of deployments, the list just grows and grows.This PR fixes both of those issues by introducing some settings:
new_containers_replay_logs
- Allows disabling of replay of logs for existing containers. This should be set totrue
when upgrading. It can later be switched back to help guarantee log delivery while Heka is not listening. Defaults totrue
container_expiry_days
- Represents the number of days to keep a container ID in the file after we last see it. Defaults to 30 days.There is some additional refactoring of the code for encapsulation and readability. Finally it adds documentation for a missing setting introduced earlier,
fields_from_labels
.