Splunk Lab

This project lets you stand up a Splunk instance in Docker on a quick and dirty basis.

But what is Splunk? Splunk is a platform for big data collection and analytics. You feed your events from syslog, webserver logs, or application logs into Splunk, and can use queries to extract meaningful insights from that data.

Quick Start!

Paste either of these on the command line:

bash <(curl -s https://raw.githubusercontent.com/dmuth/splunk-lab/master/go.sh)

bash <(curl -Ls https://bit.ly/splunklab)

...and the script will print up what directory it will ingest logs from, your password, etc. Follow the on-screen instructions for setting environment variables and you'll be up and running in no time! Whatever logs you had sitting in your logs/ directory will be searchable in Splunk with the search index=main.

If you want to see neat things you can do in Splunk Lab, check out the Cookbook section.

Also, the script will craete a directory called bin/ with some helper scripts in it. Be sure to check them out!

Useful links after starting

https://localhost:8000/ - Default port to log into the local instance. Username is admin, password is what was set when starting Splunk Lab.
Splunk Dashboard Examples - Wanna see what you can do with Splunk? Here are some example dashboards.

Features

App databoards can be stored in the local filesystem (they don't dissappear when the container exits)
Ingested data can be stored in the local filesystem
Multiple REST and RSS endpoints "built in" to provide sources of data ingestion
Integration with REST API Modular Input
Splunk Machine Learning Toolkit included
/etc/hosts can be appended to with local ip/hostname entries
Ships with Eventgen to populate your index with fake webserver events for testing.

Screenshots

These are screenshots with actual data from production apps which I built on top of Splunk Lab:

Splunk Lab Cookbook

What can you do with Splunk Lab? Here are a few examples of ways you can use Splunk Lab:

Ingest some logs for viewing, searching, and analysis

Drop your logs into the logs/ directory.
bash <(curl -Ls https://bit.ly/splunklab)
Go to https://localhost:8000/
Ingsted data will be written to data/ which will persist between runs.

Ingest some logs for viewing, searching, and analysis but DON'T keep ingested data between runs

SPLUNK_DATA=no bash <(curl -Ls https://bit.ly/splunklab)
Note that data/ will not be written to and launching a new container will cause logs/ to be indexed again.
- This will increase ingestion rate on Docker for OS/X, as there are some issues with the filesystem driver in OS/X Docker.

Play around with synthetic webserver data

SPLUNK_EVENTGEN=1 bash <(curl -Ls https://bit.ly/splunklab)
Fake webserver logs will be written every 10 seconds and can be viewed with the query index=main sourcetype=nginx. The logs are based on actual HTTP requests which have come into the webserver hosting my blog.

Adding Hostnames into /etc/hosts

Edit a local hosts file
ETC_HOSTS=./hosts bash <(curl -Ls https://bit.ly/splunklab)
This can be used in conjunction with something like Splunk Network Monitor to ping hosts that don't have DNS names, such as your home's webcam. :-)

Get the Docker command line for any of the above

Run any of the above with PRINT_DOCKER_CMD=1 set, and the Docker command line that's used will be written to stdout.

Run Splunk Lab in Development Mode with a bash Shell

This would normally be done with the script ./bin/devel.sh when running from the repo, but if you're running Splunk Lab just with the Docker image, here's how to do it:

docker run -p 8000:8000 -e SPLUNK_PASSWORD=password1 -v $(pwd)/data:/data -v $(pwd)/logs:/logs --name splunk-lab --rm -it -v $(pwd):/mnt -e SPLUNK_DEVEL=1 dmuth1/splunk-lab bash

This is useful mainly if you want to poke around in Splunk Lab while it's running. Note that you could always just run docker exec splunk-lab bash instead of doing all of the above. :-)

Splunk Apps Included

The following Splunk apps are included in this Docker image:

dmuth / splunk-lab

readme