search

outcoldman / docker-splunk-cluster

How to run cluster of Splunk Enterprise in Docker. Examples.
MIT License
32 stars 17 forks source link

readme

Table of Contents

Introduction

NOTE: I'm working at Splunk, but this is not an official Splunk images. I build them in my free time when I'm not at work. I have some knowledge about Splunk, but you should think twice before putting them in production. I run these images on my own home server just for my personal needs. If you have any issues - feel free to open a bug.

Use for learning purposes.

This repository contains set of examples how to run Splunk Enterprise cluster in Docker, including Search Head Cluster and Indexing Cluster.

The main purpose of this repository is to show how to automate Splunk Cluster deployment. Below you can find examples how to setup Cluster on Docker, Swarm Mode, Kubernetes (TODO).

Version

Based on

How it works

These examples depend on the custom image, which you can build using ./splunk-cluster/ folder. This image differs from outcoldman/splunk with just one change. It has special splunk_setup.py script, which allows to pre-configure Splunk. This script supports several commands:

Use it

Pre-requirement

You need to have a base Splunk image, you can use outcoldman/splunk as base image. Build it and tag it as splunk:latest on the machine where you will build this image.

Deploy

On docker instance

NOTE2: If you are using Docker for Mac - it allocates just 2Gb by default, not enough for this demo. Set more. Maybe 8Gb.

cd ./examples/docker

This folder has two docker-compose files. One which does not require License Master and Splunk Enterprise License docker-compose.yml and second is an extension for the first one, which adds License Master node. Makefile in this folder deals with how docker-compose needs to be invoked.

If you have Splunk Enterprise License copy it in this folder (make sure that license files have extension *.lic) and use all commands with -lm suffix.

Build image.

make build[-lm]

Deploy instances.

make deploy[-lm]

Watch for status of deployment:

To clean use

make clean[-lm]

On docker swarm

NOTE1: Splunk Enterprise License is required NOTE2: You have to use docker registry to be sure that each instance will have access to images built by you. Or you can publish image on every swarm instance manually. Specify path to your registry with environment variable SPLUNK_CLUSTER_DOCKER_IMAGE_PATH=registry.yourcompany.com/$USER

cd ./examples/docker-swarm-mode

Copy Splunk Enterprise license (if you have) in this folder (make sure that license files have extension *.lic).

Prepare swarm. This command will create 5 docker-machine instances in VirtualBox. 3 of them will be used in Docker Swarm right away, 2 can be added later

make setup

To use Swarm you need to have access to the Docker registry, specify path to registry and path to image using. If you will not specify anything it will publish image to hub.docker.com/u/$USER/splunk-cluster (create your repo at hub.docker.com)

export SPLUNK_CLUSTER_DOCKER_IMAGE_PATH=registry.yourcompany.com/$USER

Login to your registry (if it is required)

docker login registry.yourcompany.com

Build image.

make build

Publish image to your registry

make push

Deploy cluster.

make deploy

You can add two more nodes to the Swarm cluster by invoking

make setup-add-2

To clean splunk cluster (including volumes) use

make clean-all

To clean images (in case if you want to rebuild)

make clean-images

To download image on each docker instance

make download-image

To remove all docker machines use

make setup-clean

To distribute applications with Cluster Master to the Indexers

docker cp my_app $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master):/opt/splunk/etc/master-apps/
docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh chown -R splunk:splunk /opt/splunk/etc/master-apps/my_app
docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh splunk apply cluster-bundle --auth admin:changeme --answer-yes

To distribute application with SHC Deployer to SHC Members

docker cp my_app $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master):/opt/splunk/etc/shcluster/apps/
docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh chown -R splunk:splunk /opt/splunk/etc/shcluster/apps/my_app
docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh splunk apply shcluster-bundle -restart true --answer-yes -target https://shc-member-01:8089 -auth admin:changeme
docker service create \
    --name cadvisor \
    --mode global \
    --container-label splunk.cluster=cadvisor \
    --label splunk.cluster=cadvisor \
    --network splunk \
    --with-registry-auth \
    --publish 8080:8080 \
    --mount "type=bind,source=/,target=/rootfs,readonly=true" \
    --mount "type=bind,source=/var/run,target=/var/run,readonly=false" \
    --mount "type=bind,source=/sys,target=/sys,readonly=true" \
    --mount "type=bind,source=/var/lib/docker/,target=/var/lib/docker/,readonly=true" \
    $SPLUNK_CLUSTER_DOCKER_IMAGE_PATH/cadvisor \
        -storage_driver=splunk \
        -storage_driver_splunk_insecureskipverify=true \
        -storage_driver_splunk_source=cadvisor \
        -storage_driver_splunk_token=EF211A51-D6AC-4045-8CD6-F730939AC518 \
        -storage_driver_splunk_url=https://cluster-slave:8088

On kubernetes

TODO

Examples after setup

Install application on SHC using SHC Deployer

docker cp ~/Downloads/splunk_app_aws shc-deployer:/opt/splunk/etc/shcluster/apps/
docker exec shc-deployer entrypoint.sh chown -R splunk:splunk /opt/splunk/etc/shcluster/apps/
docker exec shc-deployer entrypoint.sh splunk apply shcluster-bundle -restart true --answer-yes -target https://$(docker ps --filter=label=splunk.cluster=shc-member -q | head -1):8089 -auth admin:changeme