How to maintain configuration for the Splunk instance?

[mpdude]

I am considering running Splunk in Docker to replace my current .deb based install. My expectation is that a Docker-based setup would make it much easier to upgrade to new Splunk versions in the future. Also, I don't need a particularly fancy setup for just 1-2 GB of log volume per day.

One thing, however, I don't understand yet is what the recommended approach is to keep index data and Splunk configuration out of the running image, and at best under version control (?).

My users need to be able to install additional apps, they create dashboards and alerts and have their reports configured. Additionally, we've got custom sourcetypes and made quite some modifications to the inputs.conf, props.conf and transform.conf files. Last but not least, there's SSL certificates for the web interface, Splunk-2-Splunk and HTTP Event Collector.

Could someone point me in the right direction here? Should I just mount /opt/splunk/var and /opt/splunk/etc as volumes, and can I carry those over to new versions of the Splunk image? Is there a better way or even a recommended approach documented somewhere?

Thanks for your help!

[jmervine]

@mpdude (disclaimer: I don't work for splunk)

The short answer is yes, simply mount /opt/splunk/{var,etc} in to the running container. If you're working on machine with an existing splunk install, you can do something like (untested)

$ docker run -d -p 8000:8000 \
  -v /opt/splunk/var:/opt/splunk/var -v /opt/splunk/etc:/opt/splunk/etc \
  -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=<password>' \
  splunk/splunk:latest

Be warned though, the above will mount and update those disk locations with what's running in the container, so I wouldn't do something like that without testing in development or staging installation first.

[mpdude]

Thanks for the suggestion!

However, my impression was that there are a lot of files unter /opt/splunk/etc (especially in the folders other than local/) that I would never touch.

Wouldn’t it be better if those were the versions from the image, that is those exactly matching the Splunk version?

[lephino]

@mpdude This is why the current style of splunk's docker container untar's on top of volume mounts. There is a lot of flexibility to mount virtually any of the sub apps / var/run dirs that you'd care about, and we'll untar on top to make sure you have the latest files. I will however say, we've talked with several people about the feedback, and may eventually drop to "forcing" etc/var be mounted volumes so that we can speed up start times. Only mounting etc / var right now, is your safest option.

[jmervine]

RE "there are a lot of files unter /opt/splunk/etc (especially in the folders other than local/) that I would never touch", if I'm understanding you right, that's the kind of the point. Let the splunk image worry about changing those files -- as part of the upgrade. As long as you're only changing things in "local" directories, splunk will update the defaults, but your settings will take precedence when configuration is applied.

It would be nice if there were image versions for every splunk version, and perhaps going forward there will be (starting somewhere around 7.2.0). That's a question for someone at splunk. I'd consider upgrading (maybe first) as part of moving to the docker image.

:2cents: