search

rtoma / logspout-redis-logstash

Logspout adapter for writing Docker container logs to Redis in Logstash jsonevent layout
Apache License 2.0
33 stars 28 forks source link

readme

logspout-redis-logstash

Logspout adapter for writing Docker container stdout/stderr logs to Redis in Logstash jsonevent layout.

Since v0.1.4 JSON input is supported, enabling you to add structure to your logs.

Docker image available

Logspout including this adapter is available on Docker Hub. Pull it with:

$ docker pull rtoma/logspout-redis-logstash

How to use the Docker image

$ docker run -d --name logspout \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  rtoma/logspout-redis-logstash \
  redis://<your-redis-server>?key=bla&...

Configuration

Some configuration can be passed via container environment keys.

Some can be passed via route options (e.g. logspout redis://<host>?key=foo&password=secret).

This table shows all configuration parameters:

Parameter Default Environment key Route option key
Enable debug, if set debug logging will be printed disabled DEBUG debug
Redis password, if set this will force the adapter to execute a Redis AUTH command none REDIS_PASSWORD password
Redis key, events will be pushed to this Redis list object 'logspout' REDIS_KEY key
Redis database, if set the adapter will execute a Redis SELECT command 0 REDIS_DATABASE database
Docker host, will add a docker.host=\<host> field to the event, allowing you to add the hostname of your host, identifying where your container was running (think mesos) none REDIS_DOCKER_HOST docker_host
Use Layout v0, what Logstash json format is used. With v0 JSON input support is disabled. false (meaning we use v1) REDIS_USE_V0_LAYOUT use_v0_layout
Logstash type, if set the event will get a @type property none REDIS_LOGSTASH_TYPE logstash_type
If true, will replace all "." in container labels with "_". You need to set this if you are using Elasticsearch 2.x false DEDOT_LABELS dedot_labels
Mute errors (to avoid error storm), disable by setting to other than 'true' true MUTE_ERRORS mute_errors
Redis connection timeout 100 ms CONNECT_TIMEOUT connect_timeout
Redis read timeout 300 ms READ_TIMEOUT read_timeout
Redis write timeout 500 ms WRITE_TIMEOUT write_timeout

Note on timeouts: Logspout stops tailing a container log if an adapter (like this one) takes longer than 1.0 second to process an event. That's why the sum of our default timeouts is a safe 900 ms.

JSON input support

Note: this does not work when using the Logstash v0 layout.

Since v0.1.4 JSON input is supported, enabling you to add structure to your logs. Big words, but this is what it does.

Imagine your docker application to emit a log to stdout like:

{"message":"I was very busy","items_processed":42,"elapsed_time_ms":123}

Logspout-redis-logstash will recognize this log as a JSON string and will embed the fields into the JSON document. This is how the final JSON document looks like that will be send to Redis:

{
  "@timestamp": "2016-03-30T09:54:24.587277635Z",
  "host": "ef5fe78b6a8e",
  "message": "I was very busy",
  "docker": { ... },
  "event": {
    "elapsed_time_ms": 123,
    "items_processed": 42
  }
}

What just happened?

Logtypes

Using the logtype field in the input JSON doc, allows you to control the name of the field hash. This was added to give you some control over different logtypes you may want to implement.

Usecase: imagine you have an application that handles HTTP requests and wants to emit acceslog and applicationlog events. These events are different and you want to handle them differently.

We currently support two types:

Example input JSON to illustrate this "data wrangling" feature:

{"logtype":"applog","message":"something went bOOm!","level":"ERROR","line":42,"file":"source.go"}

Which results in this JSON doc for Redis:

{
  "@timestamp": "2016-03-30T10:03:00.016733514Z",
  "host": "16cecd099d78",
  "message": "something went bOOm!",
  "docker": { ... },
  "logtype": "applog",
  "applog": {
    "file": "source.go",
    "level": "ERROR",
    "line": 42
  }
}

See what happened?

If you'd used "logtype":"accesslog" the JSON doc would have looked like:

{
  ... ,
  "logtype": "accesslog",
  "accesslog": {
    "file": "source.go",
    "level": "ERROR",
    "line": 42
  }
}

Building

Use ./build.sh <version> to build a Docker image for a tagged version. If you developing, use ./build.sh -d <version> to build a Docker image from local sources.

Contribution

Want to add features? Feel welcome to submit a pull request!

If you are unable to code, feel free to create a issue describing your feature request or bug report.

Changelog

0.1.8, 0.1.9 and 0.1.10

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

ELK integration

Try out logspout with redis-logstash adapter in a full ELK stack. A docker-compose.yml can be found in the example/ directory.

When logspout with adapter is running. Executing something like:

docker run --rm centos:7 echo hello from a container

Will result in a corresponding event in Elasticsearch. Below is a screenshot from Kibana4:

Credits

Thanks to Gliderlabs for creating Logspout!

Much thanks to all contributors.

For other credits see the header of the redis.go source file.