Solution to configure Splunk internal accounts / passwd (used as service accounts for scheduled searches)

Our Splunk clusters are configured to use LDAP or SAML authentication strategies for human users, But we use internal Splunk users to run scheduled searches or searches running as owner as:

These searches do not perform any authentication, these internal accounts are just placeholders to configure access control
LDAP service accounts are generally configured with a password expiration, and get locked or deleted if not authenticating for a period of time
We don't want these searches to be owned by a human user (at least in production), so that they don't stop working suddenly if the user leaves the company or moves to a different team

We generally define a Splunk internal service account per Splunk app, and our CI pipeline ensures that the owner of all the objects of the apps which require an owner are associated to this internal service account following a naming convention.

In our traditional environment, an ansible playbook creates these service accounts by generating the /opt/splunk/etc/passwd`` file from a Jinja template, and calls the APIservices/authentication/providers/services/_reload``` to reload it, We can configure a role for these accounts, and their (strong) password hash is generated randomly on the fly and automatically modified on any app deployment (an option to completely disable authentication would be even better but doesn't seem to exist).

The splunk-operator doesn't seem to provide the possibility to configure internal Splunk accounts for now, and managing them manually through the UI can quickly become heavy with a high cluster x app cardinality.

splunk / splunk-operator

Solution to configure Splunk internal accounts / passwd (used as service accounts for scheduled searches) #235