search

sendgridlabs / loggly-docker

Docker container for loggly (via syslog)
MIT License
44 stars 39 forks source link

Latest release (1.1) causes host to run out of disk space due to garbage in the logs #14

Closed ebr closed 9 years ago

ebr commented 9 years ago

I deployed the latest release of the Docker image yesterday, and the host ran out of disk space within minutes. It appears the loggly-docker container continuously writes (to its own logs) messages of the following format: 2015-09-29T01:10:22Z f99888cdb546 c63793e7-0278-4c05-8607-89104e669ca3[1]:, where the UUID appears to be the name of the container sending the syslog message (in my case, a logspout container). The logs are written to Loggly as well. I use https://github.com/gliderlabs/logspout to push log messages to loggly-docker over UDP, but i have a yet untested suspicion that the messages from loggly-docker container are getting caught by logspout which in turn sends them back to loggly-docker in a nightmare of recursion.

I reverted to version 1.0 for now which doesn't have this issue.

ebr commented 9 years ago

just wanted to clarify that there are several hundreds of the above-formatted messages sent to Loggly (and written to stdout) for every actual legit message logged by the logspout container.

pretty confident this has to do with https://github.com/sendgridlabs/loggly-docker/commit/70824a6a46540f68c3deccb89df4efb26f2b86a3, because: "This sends to Loggly as well as outputs to stdout (still visible in container logs)".

In my opinion, having the logs aggregated in stdout is 1) dangerous (due to disk space constraints), and 2) redundant (since this container primarily needs to serve as a forwarder to Loggly and not an aggregator). At the very least there should be a flag to turn off stdout mirroring.

I'll take a stab at a PR.

jonathan-short commented 9 years ago

Making an option to enable/disable logging to stdout sounds great. Defaulting to off sounds good to me. PR is appreciated, thanks!