search

mozilla-services / heka

DEPRECATED: Data collection and processing made easy.
http://hekad.readthedocs.org/
Other
3.4k stars 531 forks source link

Fix operational issue when upgrading to newer DockerLogInput from old one #1933

Closed relistan closed 8 years ago

relistan commented 8 years ago

We ran into some trouble when you upgrade from the existing DockerLogInput plugin to the current one on dev. That's because switching from using attach to logs on the Docker API means that we're going to pull in all the past logs for containers that in some cases have been up for quite some time. This leads to some pretty massive duplication of logs and a burying of the downstream logging system for awhile. Secondly, there is currently no way to expire entries from the sinces JSON file and so for systems with a lot of deployments, the list just grows and grows.

This PR fixes both of those issues by introducing some settings:

  1. new_containers_replay_logs - Allows disabling of replay of logs for existing containers. This should be set to true when upgrading. It can later be switched back to help guarantee log delivery while Heka is not listening. Defaults to true
  2. container_expiry_days - Represents the number of days to keep a container ID in the file after we last see it. Defaults to 30 days.

There is some additional refactoring of the code for encapsulation and readability. Finally it adds documentation for a missing setting introduced earlier, fields_from_labels.