What does Splunk Connect for Docker do?

Splunk Connect for Docker is a plug-in that extends and expands Docker's logging capabilities so that customers can push their Docker and container logs to their Splunk on-premise or cloud deployment.

Splunk Connect for Docker is a supported open source product. Customers with an active Splunk support contract receive Splunk Extension support under the Splunk Support Policy, which can be found at https://www.splunk.com/en_us/legal/splunk-software-support-policy.html.

See the Docker Engine managed plugin system documentation at https://docs.docker.com/engine/extend/ on support for Microsoft Windows and other platforms. See the Prerequisites in this document for more information about system requirements.

Prerequisites

Before you install Splunk Connect for Docker, make sure your system meets the following minimum prerequisites:

• Docker Engine: Version 17.05 or later. If you plan to configure Splunk Connect for Docker via 'daemon.json', you must have the Docker Community Edition (Docker-ce) 18.03 equivalent or later installed.
• Splunk Enterprise, Splunk Light, or Splunk Cloud version 6.6 or later. Splunk Connect for Docker plugin is not currently supported on Windows.
• For customers deploying to Splunk Cloud, HEC must be enabled and a token must be generated by Splunk Support before logs can be ingested.
• Configure an HEC token on Splunk Enterprise or Splunk Light (either single instance or distributed environment). Refer to the set up and use HTTP Event Collector documentation for more details.
• Operating System Platform support as defined in Docker Engine managed plugin system documentation.

Install and configure Splunk Connect for Docker

Step 1: Get an HTTP Event Collector token

You must configure the Splunk HTTP Event Collector (HEC) to send your Docker container logging data to Splunk Enterprise or Splunk Cloud. HEC uses tokens as an alternative to embedding Splunk Enterprise or Splunk Cloud credentials in your app or supporting files. For more about how the HTTP event collector works, see http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/UsetheHTTPEventCollector

1. Enable your HTTP Event collector: http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/HECWalkthrough#Enable_HEC
2. Create an HEC token: http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/UsetheHTTPEventCollector http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/UseHECusingconffiles

Note the following when you generate your token:

• Make sure that indexer acknowledgement is disabled for your token.
• Optionally, enable the indexer acknowledgement functionality by clicking the Enable indexer management checkbox.
• Do not generate your token using the default TLS cert provided by Splunk. The default certificates are not secure. For information about configuring Splunk to use self-signed or third-party certs, see http://docs.splunk.com/Documentation/Splunk/7.0.3/Security/AboutsecuringyourSplunkconfigurationwithSSL.
• Splunk Cloud customers must file a support request in order to have a token generated.

Step 2: Install the plugin

There are multiple ways to install Splunk Connect for Docker, Splunk recommends installing from Docker Store (option 1) to ensure the most current and stable build.

Install the Plugin from Docker Store

1. Pull the plugin from docker hub

   $ docker plugin install splunk/docker-logging-plugin:latest --alias splunk-logging-plugin

2. Enable the plugin if needed:

   $ docker plugin enable splunk-logging-plugin

   Install the plugin from the tar file

3. Clone the repository and check out release branch

   $ git clone  https://github.com/splunk/docker-logging-plugin.git

4. Create the plugin package

   $ cd docker-logging-plugin
   $ make package # this creates a splunk-logging-plugin.tar.gz

5. unzip the package

   $ tar -xzf splunk-logging-plugin.tar.gz

6. Create the plugin

   $ docker plugin create splunk-logging-plugin:latest splunk-logging-plugin/

7. Verify that the plugin is installed by running the following command:

   $ docker plugin ls

8. Enable the plugin

   $ docker plugin enable splunk-logging-plugin:latest

   Step 3: Run containers with the plugin installed

Splunk Connect for Docker continually listens for logs, but your containers must also be running so that the container logs are forwarded to Splunk Connect for Docker. The following examples describe how to configure containers to run with Splunk Connect for Docker.

To start your containers, refer to the Docker Documentation found at:

https://docs.docker.com/config/containers/logging/configure/
https://docs.docker.com/config/containers/logging/configure/#configure-the-delivery-mode-of-log-messages-from-container-to-log-driver

Examples

This sample daemon.json command configures Splunk Connect for Docker for all containers on the docker engine. Splunk Software recommends that when working in a production environment, you pass your HEC token through daemon.json as opposed to the command line.

{
  "log-driver": "splunk-logging-plugin",
  "log-opts": {
    "splunk-url": "<splunk_hec_endpoint>",
    "splunk-token": "<splunk-hec-token>",
    "splunk-insecureskipverify": "true"
  }
}

This sample command configures Splunk Connect for Docker for a single container.

$ docker run --log-driver=splunk-logging-plugin --log-opt splunk-url=<splunk_hec_endpoint> --log-opt splunk-token=<splunk-hec_token> --log-opt splunk-insecureskipverify=true -d <docker_image>

Step 4: Set Configuration variables

Use the configuration variables to configure the behaviors and rules for Splunk Connect for Docker. For example you can confiugre your certificate security or how messages are formatted and distributed. Note the following:

• Configurations that pass though docker run --log-opt are effective instantaneously.
• You must restart the Docker engine after configuring through <addr>daemon.json<addr>

How to use the variables

The following is an example of the logging options specified for the Splunk Enterprise instance. In this example:

The path to the root certificate and Common Name is specified using an HTTPS scheme to be used for verification.

$ docker run --log-driver=splunk-logging-plugin\
             --log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 \
             --log-opt splunk-url=https://splunkhost:8088 \
             --log-opt splunk-capath=/path/to/cert/cacert.pem \
             --log-opt splunk-caname=SplunkServerDefaultCert \
             --log-opt tag="{{.Name}}/{{.FullID}}" \
             --log-opt labels=location \
             --log-opt env=TEST \
             --env "TEST=false" \
             --label location=west \
             <docker_image>

Required Variables

Optional Variables

Advanced options - Environment Variables

To overwrite these values through environment variables, use docker plugin set =. For more information, see https://docs.docker.com/engine/reference/commandline/plugin_set/ .

Message formats

There are three logging plug-in messaging formats set under the optional variable splunk-format:

• inline (this is the default format)
• json
• raw

The default format is inline, where each log message is embedded as a string and is assigned to "line" field. For example:

// Example #1
{
    "attrs": {
        "env1": "val1",
        "label1": "label1"
    },
    "tag": "MyImage/MyContainer",
    "source":  "stdout",
    "line": "my message"
}

// Example #2
{
    "attrs": {
        "env1": "val1",
        "label1": "label1"
    },
    "tag": "MyImage/MyContainer",
    "source":  "stdout",
    "line": "{\"foo\": \"bar\"}"
}

When messages are JSON objects, you may want to embed them in the message sent to Splunk.

To format messages as json objects, set --log-opt splunk-format=json. The plug-in will try to parse every line as a JSON object and embed the json object to "line" field. If it cannot parse the message, it is sent inline. For example:

//Example #1
{
    "attrs": {
        "env1": "val1",
        "label1": "label1"
    },
    "tag": "MyImage/MyContainer",
    "source":  "stdout",
    "line": "my message"
}

//Example #2
{
    "attrs": {
        "env1": "val1",
        "label1": "label1"
    },
    "tag": "MyImage/MyContainer",
    "source":  "stdout",
    "line": {
        "foo": "bar"
    }
}

If --log-opt splunk-format=raw, each message together with attributes (environment variables and labels) and tags are combined in a raw string. Attributes and tags are prefixed to the message. For example:

MyImage/MyContainer env1=val1 label1=label1 my message
MyImage/MyContainer env1=val1 label1=label1 {"foo": "bar"}

Troubleshooting

If your Splunk Connector for Docker does not behave as expected, use the debug functionality and then refer to the following tips included in output.

Enable Debug Mode to find log errors

Plugin logs can be found as docker daemon log. To enable debug mode, export environment variable LOGGIN_LEVEL=DEBUG in docker engine environment. See the Docker documentation for information about how to enable debug mode in your docker environment: https://docs.docker.com/config/daemon/

Use the debugger to check your debug the Splunk HEC connection

Check HEC endpoint accessibility Docker environment. If the endpoint cannot be reached, debug logs are not sent to Splunk, or the logs or will buffer and drop as they roll off the buffer.

Test HEC endpoint is accessible
$ curl -k https://<ip_address>:8088/services/collector/health
{"text":"HEC is healthy","code":200}

Check your HEC configuration for clusters

If you are using an Indexer Cluster, the current plugin accepts a single splunk-url value. We recommend that you configure a load balancer in front of your Indexer tier. Make sure the load balancer can successfully tunnel the HEC requests to the indexer tier. If HEC is configured in an Indexer Cluster environment, all indexers should have same HEC token configured. See http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/UsetheHTTPEventCollector.

Check your heavy forwarder connection

If you ae using a heavy forwarder to preprocess the events (e.g: funnel multiple log lines to a single event), make sure that the heavy forwarder is properly connecting to the indexers. To troubleshoot the forwarder and receiver connection, see: https://docs.splunk.com/Documentation/SplunkCloud/7.0.0/Forwarding/Receiverconnection.

Check the plugin's debug log in docker

Stdout of a plugin is redirected to Docker logs. Such entries have a plugin= suffix.

To find out the plugin ID of Splunk Connect for Docker, use the command below and look for Splunk Logging Plugin entry.

# list all the plugins
$ docker plugin ls

Depending on your system, location of Docker daemon logging may vary. Refer to Docker documentation for Docker daemon log location for your specific platform. Here are a few examples:

• Ubuntu (old using upstart ) - /var/logging/upstart/docker.logging
• Ubuntu (new using systemd ) - sudo journalctl -fu docker.service
• Boot2Docker - /var/logging/docker.logging
• Debian GNU/Linux - /var/logging/daemon.logging
• CentOS - /var/logging/daemon.logging | grep docker
• CoreOS - journalctl -u docker.service
• Fedora - journalctl -u docker.service
• Red Hat Enterprise Linux Server - /var/logging/messages | grep docker
• OpenSuSE - journalctl -u docker.service
• OSX - ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/logging/d‌ocker.logging
• Windows - Get-EventLog -LogName Application -Source Docker -After (Get-Date).AddMinutes(-5) | Sort-Object Time, as mentioned here.