Logspout integration for logging through Docker

[sandys]

A lot (most?) deployments of applications happen through docker. These use frameworks like Docker Swarm or Kubernetes to deploy. The deployment has little awareness of the underlying hardware/OS.

Any logging solution must integrate into Docker deployment frameworks.

A popular/generic solution is Logspout - https://github.com/gliderlabs/logspout

It is now adopted by a lot of log providers - https://github.com/logdna/logspout and https://www.loggly.com/docs/docker-logging-logspout/

[cyriltovena]

Does the docker driver is enough or does logsprout has some other interesting features ?

[megamorf]

@Kuqd Its main advantage is that it keeps the regular docker log output intact whereas specifying a docker logging driver per container or host breaks docker logs entirely. It can also use multiple log "adapters" in parallel, e.g. send logs to graylog and syslog. In addition, it also offers routes to direct certain events to a different log system based on container names/labels.

Unfortunately, gliderlabs have significantly neglected its user base (i.e. lots of uncommented issues and pending pull requests). However, it'd still be a great addition to have since this would allow us to test loki while still sending all of our logs to our present log collector.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[slim-bean]

This issue has a lot of thumbs up, is this something people would still like to see?

[trevorwhitney]

Does @jeschkies work [#4911] on adding a docker target to promtail solve this problem?

[megamorf]

@trevorwhitney Nope, at least for now the promtail docker target implementation is incomplete. Docker will split up long events logged by the container application into multiple lines but promtail fails to reassemble those. It essentially results in broken JSON objects arriving in Loki.

See https://github.com/grafana/loki/issues/2281

Edit: Oh, I see. Fetching the logs from the API similar to how logspout does it would work around the line break issue that you have to handle when parsing the container logs. It'd be great if container labels and host labels are properly attached to the event in that implementation.

[megamorf]

This issue has a lot of thumbs up, is this something people would still like to see?

And yes, this is something that I'm still interested to see. Right now I have logspout sending container events to Graylog and would like to send those to Loki in parallel. That would enable me to prepare for shifting everything to loki seamlessly.

[trevorwhitney]

@megamorf I believe @jeschkies' implementation correctly adds those labels, but I haven't looked closely enough at it to know for sure.

[jeschkies]

@megamorf, yes the container labels are added. I'm not sure what you mean with host labels. WE use the service discovery from https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config.

[megamorf]

@megamorf, yes the container labels are added. I'm not sure what you mean with host labels. WE use the service discovery from https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config.

You can also add common labels to the Docker daemon.json, see:

• https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file
• https://docs.docker.com/config/containers/logging/configure/

A typical use case is to assign certain Docker hosts a label for a cluster, region or environment.

[cstyan]

@megamorf We're currently reevaluating promtails position as a project within Grafana Labs. Internally we're actually using the Agent for both metrics and logs collection at this point. While we haven't made a formal decision yet, we expect in the near future that all new feature work will be done in the Agent's log collection pipelines rather than in Promtail. Would you be interested in opening an issue there for this feature?