Docker container fails to start because web.conf is busy

[flyattimes]

After trying to upgrade from 9.0.5 to 9.2.1, my docker container wont boot anymore. The error is

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 16] Device or resource busy: b'/opt/splunk/etc/system/local/.ansible_tmpcxrl7u_bweb.conf' -> b'/opt/splunk/etc/system/local/web.conf'

fatal: [localhost]: FAILED! => {
    "changed": false
}

Now is suppose that this is because of this commit, which makes chages to web.conf: https://github.com/splunk/splunk-ansible/commit/aafd0198c91e333cd09e63f8498854bab082a762

I also need to make changes to web.conf (mainly caching for development purposes):

[settings]
cacheEntriesLimit = 0
cacheBytesLimit = 0
crossOriginSharingPolicy = https://localhost:8090
crossOriginSharingHeaders = *

I tried making the changes using the entrypoint, which fails because of permission denied error to web.conf

May I suggest introducing environment variables that allow for setting the above headers?

Also, I would greatly appreciate any hint on how to get a temporary fix for this.

[loucadufault]

I also encountered this (using docker splunk, trying to bind a server.conf file as volume to the image).

I realized the proper way to add conf files is using the defaut.yml file as described in the docs: https://github.com/splunk/splunk-ansible/blob/develop/docs/advanced/default.yml.spec.md#configuration-files

So in your example:

default.yml

splunk:
  conf:
    - key: web
      value:
        directory: /opt/splunk/etc/system/local
        content:
          settings:
            cacheEntriesLimit: 0
            [...etc]

Then bind it to the image in the docker config (e.g. for docker-compose.yml):

    volumes:
      - ./splunk/default.yml:/tmp/defaults/default.yml

[flyattimes]

@loucadufault thank you very much. That helped me alot. Now the next problem I am running into is that the set_as_hec_receiver playbook fails:

`POST/services/data/inputs/http/httpadmin****8089{'disabled': '1', 'enableSSL': '0', 'port': '8088', 'serverCert': '', 'sslPassword': ''}NoneNoneNone;;; AND excep_str: No Exception, fail

Unauthorized

`

I can see that my admin password is set using env. Not sure if this is related to another issue. Will update if I find anything.

[flyattimes]

Things are up and running now. The admin user I had configured using the /etc/passwd file, which I had tried to move to default.yml as well. Since default.yml does not appear to allow to represent the values found in /etc/passwd, I had to go back to volume mounting it.

This is in short what it looks like: docker-compose.yml

  splunk_stand_alone:
    networks:
      splunknet:
        aliases:
          - splunk_stand_alone
    image: splunk/splunk:9.2.1
    hostname: splunk_stand_alone
    container_name: splunk_stand_alone
    environment:
       - SPLUNK_START_ARGS=--accept-license
       - SPLUNK_DEPLOYMENT_SERVER=splunk_deployment_server
       - SPLUNK_PASSWORD=${SPLUNK_PASSWORD}
       - SPLUNK_LICENSE_URI=/tmp/splunk.License
       - SPLUNK_TAIL_FILE=/opt/splunk/var/log/splunk/splunkd.log     
       - ANSIBLE_EXTRA_FLAGS=--extra-vars "@/tmp/defaults/default.yml" -vvvv
    volumes:
      - ./default.yml:/tmp/defaults/default.yml
      - ./splunk-license/splunk.License:/tmp/splunk.License
      - ../splunk-apps/test_env_settings/etc/passwd:/opt/splunk/etc/passwd:rw
      - ../splunk-apps/test_env_settings/indexes.conf:/opt/splunk/etc/system/local/indexes.conf:rw
      - ../splunk-apps/test_env_settings/inputs.conf:/tmp/inputs.conf:rw

As far as I can see, it is also not possible to move inputs.conf to default.yml (at least for my use case!) as I could not see how to get the stanza name splunktcp-ssl:9997 to work as a key, value pair. Hence inputs has to be mangled via volume mounts

default.yml (only non-default sections)

    - key: web
      value:
        directory: /opt/splunk/etc/system/local
        content:
          settings:
            cacheEntriesLimit: 0
            cacheBytesLimit: 0
    - key: server
      value:
        directory: /opt/splunk/etc/system/local
        content:
          sslConfig:
            sslRootCAPath: /opt/splunk/....
    - key: authorize
      value:
        directory: /opt/splunk/etc/system/local
        content:
          default:
            ...
          role_a:
            importRoles: default
            srchIndexesAllowed: abc
          role_b:
            importRoles: default
            srchIndexesAllowed: xyz