App Framework: Can't configure a lot of stuff using configuration apps

[marcusschiesser]

Please select the type of request

Bug

Tell us more

Describe the request Using the App Framework, the idea is to use configuration apps to configure the Splunk instance. Unfortunately, a lot of configurations are not possible just by using configuration apps. Also, I couldn't find any documentation of what is configurable via an app and what is not, so it's a lot of trial and error.

Examples of what is not configurable:

1. server.conf changes, e.g. minFreeSpace from the [diskUsage] stanza
2. In user-prefs.conf, setting the default_namespace for a role, see https://splunk-usergroups.slack.com/archives/CDE623ETD/p1658159047934719
3. Changing the permissions of an existing app via local.meta file, e.g. for splunk-dashboard-studio
4. Disabling an existing app for security reasons, e.g. splunk_secure_gateway

Expected behavior

• Some of these examples are understandable (e.g. server.conf changes) - they should be documented in https://docs.splunk.com/Documentation/Splunk/8.2.7/Admin/Serverconf and be configurable in the long run. My current workaround is to configure them via default.xml, but as default.xml is needed for the #657 workaround, it's a very bad separation of concerns.
• Some are not understandable and should be considered bugs (e.g. user-prefs.conf) - my workaround to configure them would be via a post init hook as proposed in #875
• Changing and disabling existing apps should be generally supported

Splunk setup on K8S Splunk Enterprise 8.2.7 with Operator 1.1.0

Reproduction/Testing steps

• Depends on example, e.g. deploy app as described in https://splunk-usergroups.slack.com/archives/CDE623ETD/p1658159047934719

K8s environment EKS with K8S 1.21

[sgontla]

@marcusschiesser , the App Framework doesn't inspect the contents of the app package. it uses the app package file(.tgz/.spl), and installs it on Splunk instance in case of local scope. For cluster scope, the app package is included a part of the cluster apps location, and triggers a bundle push. For any given config lines, whatever is the default Splunk behavior( in dealing with any config stanza) with respect to the app install OR bundle push, same applies for the Operator's app framework.

[marcusschiesser]

@sgontla sure I understand that, but from the customer's point of view, it doesn't matter by which component the issues above are caused. If I manually install a Splunk instance, I won't run into these issues, but if I use the App Framework for configuration I will. So from the customer's perspective, it's an error that happens because of using the App Framework.

[marcusschiesser]

My current workaround for this issue is to deploy another pod in the same namespace as the Splunk instance. That pod is periodically configuring the Splunk instance using the REST API via curl.

For example, to configure the default app of a role user to My-APP (for the Splunk instance splunk-mysplunk-standalone-service), I am using this pod:

apiVersion: v1
kind: Pod
metadata:
  name: splunk-configurator
spec:
  containers:
  - name: splunk-configurator
    image: curlimages/curl:7.85.0
    command: ["/bin/sh"]
    args: ["-c", "while true; do curl -k -u admin:$(cat /mnt/splunk-secrets/password) https://splunk-mysplunk-standalone-service:8089/services/authorization/roles/user --data 'defaultApp=My-APP'; sleep 30; done"]
    volumeMounts:
    - mountPath: /mnt/splunk-secrets
      name: mnt-splunk-secrets
  volumes:
  - name: mnt-splunk-secrets
    secret:
      secretName: splunk-default-secret          

Note: This hack is reusing the secret splunk-default-secret from the operator

[satellite-no]

This issue is in the wrong place... Its not an issue with app-framework.. app-framework just installs apps you tell it too.

If you actaully want that changed submit that to Splunk proper not the splunk-operator repo.

[satellite-no]

Also an easier more sustainable IaC way to work around this would be something like this.

https://github.com/splunk/splunk-ansible/blob/develop/docs/advanced/default.yml.spec.md#configuration-files

[marcusschiesser]

Also an easier more sustainable IaC way to work around this would be something like this. https://github.com/splunk/splunk-ansible/blob/develop/docs/advanced/default.yml.spec.md#configuration-files

Sure, you can use the default.yml configuration, but I already use it to inject passwords from K8S secrets (see #657). I don't want to mix both concerns (credentials and non-credentials), so I suggest here the alternative approach as a workaround.

Yes, I agree it's not a problem of the splunk-operator per se, but it only occurs if someone is using solely configuration apps for Splunk configuration. Most people would just change the configuration file and restart Splunk. So my assumption is that this issue only happens with using the splunk-operator. But I am happy to re-post this issue somewhere else on GitHub if someone sends me the repo link.