containers are not mounting the var volume

[jmaas]

Hi, I'm trying to get docker-splunk running and am having issues with the var volume which doesn't get mounted inside the container. It is working for the etc volume however.

This is my compose file, I've tried this with Docker CE (latest on Ubuntu 22.04) in either a stack/swarm or just with docker-compose:

version: "3.6"

networks:
  splunknet:

volumes:
  sh1-etc:
  sh1-var:
  cm1-etc:
  cm1-var:
  idx1-etc:
  idx1-var:
  idx2-etc:
  idx2-var:
  idx3-etc:
  idx3-var:

services:
  sh1:
    networks:
      splunknet:
        aliases:
          - sh1
    image: splunk/splunk:latest
    hostname: sh1
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_INDEXER_URL=idx1,idx2,idx3
      - SPLUNK_SEARCH_HEAD_URL=sh1
      - SPLUNK_CLUSTER_MASTER_URL=cm1
      - SPLUNK_ROLE=splunk_search_head
    ports:
      - 8000
      - 8089
    volumes:
      - ./default.yml:/tmp/defaults/default.yml
      - sh1-etc:/opt/splunk/etc
      - sh1-var:/opt/splunk/var

  cm1:
    networks:
      splunknet:
        aliases:
          - cm1
    image: splunk/splunk:latest
    command: start
    hostname: cm1
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_INDEXER_URL=idx1,idx2,idx3
      - SPLUNK_SEARCH_HEAD_URL=sh1
      - SPLUNK_CLUSTER_MASTER_URL=cm1
      - SPLUNK_ROLE=splunk_cluster_master
    ports:
      - 8000
      - 8089
    volumes:
      - ./default.yml:/tmp/defaults/default.yml
      - cm1-etc:/opt/splunk/etc
      - cm1-var:/opt/splunk/var

  idx1:
    networks:
      splunknet:
        aliases:
          - idx1
    image: splunk/splunk:latest
    command: start
    hostname: idx1
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_INDEXER_URL=idx1,idx2,idx3
      - SPLUNK_SEARCH_HEAD_URL=sh1
      - SPLUNK_CLUSTER_MASTER_URL=cm1
      - SPLUNK_ROLE=splunk_indexer
    ports:
      - 8000
      - 8089
    volumes:
      - ./default.yml:/tmp/defaults/default.yml
      - idx1-etc:/opt/splunk/etc
      - idx1-var:/opt/splunk/var

  idx2:
    networks:
      splunknet:
        aliases:
          - idx2
    image: splunk/splunk:latest
    command: start
    hostname: idx2
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_INDEXER_URL=idx1,idx2,idx3
      - SPLUNK_SEARCH_HEAD_URL=sh1
      - SPLUNK_CLUSTER_MASTER_URL=cm1
      - SPLUNK_ROLE=splunk_indexer
    ports:
      - 8000
      - 8089
    volumes:
      - ./default.yml:/tmp/defaults/default.yml
      - idx2-etc:/opt/splunk/etc
      - idx2-var:/opt/splunk/var

  idx3:
    networks:
      splunknet:
        aliases:
          - idx3
    image: splunk/splunk:latest
    command: start
    hostname: idx3
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_INDEXER_URL=idx1,idx2,idx3
      - SPLUNK_SEARCH_HEAD_URL=sh1
      - SPLUNK_CLUSTER_MASTER_URL=cm1
      - SPLUNK_ROLE=splunk_indexer
    ports:
      - 8000
      - 8089
    volumes:
      - ./default.yml:/tmp/defaults/default.yml
      - idx3-etc:/opt/splunk/etc
      - idx3-var:/opt/splunk/var

This is what df shows me inside a running container:

jmaas@srv1:~/stacks/splunk$ docker container exec 6e96bb385343 df -h
Filesystem                             Size  Used Avail Use% Mounted on
overlay                                500G  5.8G  495G   2% /
tmpfs                                   64M     0   64M   0% /dev
shm                                     64M  176K   64M   1% /dev/shm
/dev/mapper/ubuntu--vg-lv_docker_base  500G  5.8G  495G   2% /etc/hosts
/dev/mapper/ubuntu--vg-lv_root          16G  6.4G  9.7G  40% /tmp/defaults/default.yml
/dev/mapper/ssd--vg-lv_docker_volumes  1.9T   22G  1.8T   2% /opt/splunk/etc
tmpfs                                   32G     0   32G   0% /proc/acpi
tmpfs                                   32G     0   32G   0% /proc/scsi
tmpfs                                   32G     0   32G   0% /sys/firmware

Any obvious mistakes? Steps to troubleshoot?

[jmaas]

Creating persistent storage

jmaas@srv1:~/stacks/splunk$ docker volume create splunk-etc
splunk-etc
jmaas@srv1:~/stacks/splunk$ docker volume create splunk-var
splunk-var

Starting the container:

jmaas@srv1:~/stacks/splunk$ docker run -d --name splunk -v splunk-etc:/opt/splunk/etc -v splunk-var:/opt/splunk/var -v ./default.yml:/tmp/defaults/default.yml -e SPLUNK_START_ARGS=--accept-license -p 8000:8000 splunk/splunk:latest
955504c4f731e59432ad6beebaf4075c82fa107e2689895d626394c9e91ab47d

It's running and done running the Ansible playbooks

jmaas@srv1:~/stacks/splunk$ docker ps|grep splunk
955504c4f731   splunk/splunk:latest   "/sbin/entrypoint.sh…"   2 minutes ago   Up 2 minutes (healthy)   8065/tcp, 8088-8089/tcp, 8191/tcp, 9887/tcp, 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 9997/tcp   splunk

Let's see what df has to tell us:

jmaas@srv1:~/stacks/splunk$ docker container exec 955504c4f731 df -h
Filesystem                             Size  Used Avail Use% Mounted on
overlay                                500G  5.7G  495G   2% /
tmpfs                                   64M     0   64M   0% /dev
shm                                     64M     0   64M   0% /dev/shm
/dev/mapper/ubuntu--vg-lv_docker_base  500G  5.7G  495G   2% /etc/hosts
/dev/mapper/ssd--vg-lv_docker_volumes  1.9T   14G  1.9T   1% /opt/splunk/var
/dev/mapper/ubuntu--vg-lv_root          16G  6.4G  9.7G  40% /tmp/defaults/default.yml
tmpfs                                   32G     0   32G   0% /proc/acpi
tmpfs                                   32G     0   32G   0% /proc/scsi
tmpfs                                   32G     0   32G   0% /sys/firmware

Now, etc is not mounted?

[jmaas]

Okay, it seems this is just a "presentation" issue related to docker:

jmaas@srv1:~/stacks/splunk$ docker container inspect 955504c4f731 |jq ".[].Mounts"
[
  {
    "Type": "bind",
    "Source": "/home/jmaas/stacks/splunk/default.yml",
    "Destination": "/tmp/defaults/default.yml",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "volume",
    "Name": "splunk-etc",
    "Source": "/var/lib/docker/volumes/splunk-etc/_data",
    "Destination": "/opt/splunk/etc",
    "Driver": "local",
    "Mode": "z",
    "RW": true,
    "Propagation": ""
  },
  {
    "Type": "volume",
    "Name": "splunk-var",
    "Source": "/var/lib/docker/volumes/splunk-var/_data",
    "Destination": "/opt/splunk/var",
    "Driver": "local",
    "Mode": "z",
    "RW": true,
    "Propagation": ""
  }
]

I think I've gone down the wrong rabbit hole :)

[jmaas]

Okay, it's df behaviour, this command shows me the expected output:

jmaas@srv1:~/stacks/splunk$ docker container exec 955504c4f731 df -a
Filesystem                             1K-blocks     Used  Available Use% Mounted on
overlay                                524032000  5905968  518126032   2% /
proc                                           0        0          0    - /proc
tmpfs                                      65536        0      65536   0% /dev
devpts                                         0        0          0    - /dev/pts
sysfs                                          0        0          0    - /sys
cgroup                                         0        0          0    - /sys/fs/cgroup
mqueue                                         0        0          0    - /dev/mqueue
shm                                        65536        0      65536   0% /dev/shm
/dev/mapper/ubuntu--vg-lv_docker_base  524032000  5905968  518126032   2% /etc/resolv.conf
/dev/mapper/ubuntu--vg-lv_docker_base  524032000  5905968  518126032   2% /etc/hostname
/dev/mapper/ubuntu--vg-lv_docker_base  524032000  5905968  518126032   2% /etc/hosts
/dev/mapper/ssd--vg-lv_docker_volumes 1952559608 14418004 1938141604   1% /opt/splunk/var
/dev/mapper/ubuntu--vg-lv_root          16766976  6618952   10148024  40% /tmp/defaults/default.yml
/dev/mapper/ssd--vg-lv_docker_volumes 1952559608 14418004 1938141604   1% /opt/splunk/etc
proc                                           0        0          0    - /proc/bus
proc                                           0        0          0    - /proc/fs
proc                                           0        0          0    - /proc/irq
proc                                           0        0          0    - /proc/sys
proc                                           0        0          0    - /proc/sysrq-trigger
tmpfs                                   32788880        0   32788880   0% /proc/acpi
tmpfs                                      65536        0      65536   0% /proc/kcore
tmpfs                                      65536        0      65536   0% /proc/keys
tmpfs                                      65536        0      65536   0% /proc/timer_list
tmpfs                                   32788880        0   32788880   0% /proc/scsi
tmpfs                                   32788880        0   32788880   0% /sys/firmware

Closing ticket! Sorry for the noise here :)