This app allows you to back up and use version control to manage your Splunk knowledge objects, such as saved searches and macros.
Splunk (as of the time of writing in January 2019) has no native ability to use version control on its knowledge objects. This can lead to issues where an object is accidentally changed or deleted and there is no way to restore them beyond using OS-level backups, which are difficult to use in a search head cluster.
The app uses two modular inputs to back up and restore configurations, Splunk Version Control Backup (or splunkversioncontrol_backup) and and Splunk Version Control Restore (or splunkversioncontrol_restore).
The backup portion of the app provides a Splunk modular input with the ability to serialize various Splunk knowledge objects into JSON format, which is then stored in a remote git repository and tagged based on each change to the backup.
These two inputs do not have to be on the same machine, however, they must be pointing to the same git repository and the gitTempDir must be unique on the filesystem if sharing the same machine.
The restore portion provides a Splunk modular input and a dashboard (SplunkVersionControl Restore) that can be used to request the restoration of a knowledge object.
Use the SplunkVersionControl Restore dashboard to request that a knowledge object be restored to a prior version. You must be the author of the knowledge objects you wish to restore, or have the admin role. The application with the knowledge object in it must still exist on the Splunk server.
There are two unique dashboards with two different restoration methods, the original version is described below: When a knowledge object restore is requested the dashboard (SplunkVersionControl Restore) outputs the knowledge object information to a lookup with the definition splunkversioncontrol_restorelist. The modular input then triggers the restore based on the contents of this lookup, the modular input either creates or updates the knowledge object with the requested git tag, or logs the failure to find the object in the logs.
Note that the above option is the option used with Splunk Cloud, the below option can be used on on-prem instances...
The newer dynamic version follows a similar process, but instead of adding the knowledge object restore information to a lookup file it runs a Splunk custom command postversioncontrolrestore that hits a REST endpoint on either a local or a remote server.
The REST endpoint then performs a few functions:
splunkversioncontrol_restore input stanza namesplunkversioncontrol_restore input stanza and runs a remote query against itSplunk Version Control Audit Query POSTsplunk_vc_timeout macro)The ability to restore/create configuration opens up a few obvious issues:
To address these issues, a report named "SplunkVersionControl Audit Query" runs a query against the audit logs to determine if the lookup was updated by the saved search "SplunkVersionControl AddToLookup". This audit query returns a username and a time (it looks back/forwards one second from when the lookup was created).
The restoration script then validates that the username entered in the lookup file and the time match those found in the audit log. If they do not match then the restoration is rejected.
If you are using the dynamic version of the restore dashboard (custom command postversioncontrolrestore, an alternative report named "Splunk Version Control Audit Query POST" runs to check the audit logs, this report determines if the restoration request was made by the user in question. The report returns 0 or more results and if it returns results for the particular user, the restore proceeds.
Due to the above there is the possibility that multiple users may trigger a restore while a restore is in progress, a kvstore is used to prevent this from occurring and an additional restore attempt when the restore process is in progress results in an error message to try again.
If a user attempts to restore the objects of another user, or attempts to restore the objects as a different user, this is allowed if the user has the admin role (which is determined by the saved search "SplunkVersionControl CheckAdmin"). You can change this behaviour if you wish by changing this report...
A custom command named postversioncontrolrestore and the accompanying dashboard splunkversioncontrolrestore_dynamic were created for this purpose in version 1.0.7
However this version wil not work in Splunk Cloud as it would require connectivity to an on-prem instance that can perform the backup/restore modular input functions
The following assumptions are made:
No. However, the backup/restore modular input must have access to its own git temporary directory on the OS filesystem, the temporary directory should be unique for both backup and restore operations
During the first run of the script (at which point the lookup file is empty) all applications and all objects will be backed up.
During each subsequent run of the script, if an application is found in the Splunk system but not on the filesystem then the backup of all objects within that application will occur
Otherwise an incremental backup of knowledge objects occurs (see below)
There are two searches used to determine "what" has changed since the last run of the modular input:
These two searches are passed in an epoch value, they then return a list of applications and the type of config that has changed.
For example if application search has had macros changed, then all macros in the search app will be backed up, however the savedsearches/dashboards/et cetera will not be backed up unless at least one of them in that app has changed.
Each backup run that results in more than 0 file changes will auto-commit all changes into git and tag with the current date/time including the minute of the hour. This will create tags such as 2019-01-10_0136, these tags can later be used to "restore from" in the "SplunkVersionControl Restore" dashboard.
The tags are recorded by outputting the tag list into the lookup definition splunkversioncontrol_taglist within the app, this same lookup definition is read by the dashboard to list available tags to restore from.
The restoration is based on a git tag, the relevant tag is checked out on the filesystem after running a git checkout master; git pull command.
Once checked out, the app/user/global directories are checked (depending on which scope was requested) to see if there is a relevant config item to restore, if found the remote object is either updated with the contents of git or created if it does not exist. By default the knowledge object is created with the same username that was in the backup, however there is an option on the SplunkVersionControl Restore dashboard to override the user on restoration, this is only able to be done by a user with an admin role.
splunkversioncontrol_globalexclusionlist, this lookup definition records a list of excluded applicationssplunkversioncontrol_restorelist, this lookup definition records what must be restored by the restore modular input (this is used by the non-dynamic dashboard)splunkversioncontrol_taglist, this lookup definition records the tags available in gitOn a Linux-based system
/opt/splunk/var/log/splunk/splunkversioncontrol_restore.log -- this log will contain information about the splunk restore modular input/opt/splunk/var/log/splunk/splunkversioncontrol_backup.log -- this log will contain information about the splunk backup modular input/opt/splunk/var/log/splunk/splunkversioncontrol_postversioncontrolrestore.log -- this log contains information about the | postversioncontrol command/opt/splunk/var/log/splunk/splunkversioncontrol_rest_restore.log -- log log contains information about hits to the REST endpoint /services/splunkversioncontrol_rest_restoreOr the internal index which also has these log files with the sourcetype splunkversioncontrol
splunkversioncontrol_restore_dynamic dashboard the macros splunk_vc_name, splunk_vc_url, splunk_vc_timeout may need customisation to match your environment. In particular the splunk_vc_name assumes you have called your SplunkVersionControlRestore modular input "Prod". See the macros section of this document for more informationsplunkversioncontrol_restore_dynamic dashboard then you do not need to set a run interval, if you are using the older method you want to run this on an interval to check if the lookup file has been updated and if a restore is required...splunkversioncontrol_restore dashboard, the dynamic dashboard cannot be used in SplunkCloudgithub and other websites may offer to initialize the repository for you, if they do not the steps are usually similar to:
There are also many online resources to help with learning git
password:<name in passwords.conf> and the app will attempt to find the password in your passwords.conf filesplunkversioncontrol_globalexclusionlist lookup file, if an application name is specified only the application is backed up, can be a comma separted listsplunk_vc_ko_query (defaults to false)run_ko_query to be true, defaults to false)"More settings"
_audit index along with access to the REST endpoint for checking if users are admins. Finally this is the user used to restore a knowledge objectpassword:<name in passwords.conf> and the app will attempt to find the password in your passwords.conf file"SplunkVersionControl Audit Query" to confirm the restore request came from the search head, this must be in Splunk format (-10m, -1h or similar)"More settings"
To get passwords into or out of the passwords.conf you may wish to use REST storage/passwords Manager for Splunk
The context of the application name (default of SplunkVersionControl) will be checked first for the password, if that fails a query to all contexts /-/-/ will occur, realms will be ignored, only the name of the password is used for searching so any realm (or lack of realm) will work for storing the password
srcURL - so this is the remote port of the Splunk Cloud instance or localhost, for example: https://mycloudinstance.splunkcloud.com:8089
srcUsername - as you'd expect, the username to login via REST API
srcPassword - you can specify it in plaintext or if you do something like password:splunkversioncontrol_user, then the splunkversioncontrol_user should be in passwords.conf (I use REST storage/passwords Manager for Splunk
to add/remove passwords from passwords.conf but the command line works too)
gitTempDir - I use /tmp/git_backup but the location can be any empty directory
gitRepoURL - I use https://myuser:password:myuser_token@git.tools.company.om/scm/splunk/ko_automated_backup.git, this can be an SSH-based or a HTTPS-based repo
In the above the myuser_token again exists in passwords.conf, again tokens work, SSH-based URL's work too! Effectively this the URL you would use with "git clone"
git_name - SVC Automation
git_email - gareth...@company.com
git_proxy - if you use a proxy to access git, for example http://proxy:8080, this is optional
I do not use a proxy to access my Splunk instances but if you cannot access to the Splunk cloud instance without a proxy then set the "proxy" setting.
Finally, please tick file_per_ko that's a nicer way to store the objects in git.
I also tick disable_git_ssl_verify and I have sslVerify - false due to some issues with ssl validation
The following macros exist and are relate to the splunkversioncontrol_restore_dynamic dashboard
splunk_vc_name - this macro is the name of the splunkversioncontrol_restore modular input name on the remote (or local) system where the restore occurssplunk_vc_url - this macro is the URL endpoint of the remote system, defaults to https://localhost:8089/services/splunkversioncontrol_rest_restore, you will need to change this if you have a remote instance performing the backup/restore operations, for example if you are on a search head cluster splunk_vc_timeout - this is the time delay between triggering the remote command and waiting for the _audit index to catchup with a log entry to advise the command was run, if set too short the restore may fail because the | postversioncontrolrestore search has not appeared in the _audit index yetsslVerify - defaults to "False", this can be set to the location of a CA file to be used by the python requests library to validate the SSL certificates in userequestingAddress - by default the REST endpoint splunkversioncontrol_rest_restore will make a HTTPS call back to the calling IP address, this overrides the address to call back, the default of False results in a call back to the requesting IP address which is used in most use cases splunk_vc_ko_query, should be configured to point to an appname:searchname, the default is splunk_kom:splunk_vc_kom_audit_summaryIf run_ko_query is configured, then the app will attempt to trigger the savedsearch configured by the macro splunk_vc_ko_query
The macro should be in the format appcontext:savedsearchname
By default this is configured to splunk_kom:splunk_vc_kom_audit_summary and was tested against version 1.0.26 of the Knowledge Object Overview App for Splunk (kom) application
Note that the savedsearch splunk_vc_kom_audit_summary is included in the Splunk version control application but will need to be moved into the splunk_kom app context to work as expected, or you can make your own search if preferred.
Since the output is from the modular input, the output will default to the sourctype splunkversioncontrol_backup and will appear in the main index (you can change this in more settings)
Finally, the run_ko_diff option if configured in addition to the run_ko_query will run a git diff of HEAD~1 and include that in the output of the modular input (and will therefore be indexed into Splunk)
In some Linux OS distributions an error similar to OPENSSL_1.0.0 not found may appear, os.unsetenv('LD_LIBRARY_PATH') appears to fix this however AppInspect does not allow modification of OS environment variables.
If you have this issue please add this into the python files to workaround the problem as required Refer to this issue on github for more details
Note that you can run this from the command line if the logs are not getting populated:
splunk cmd splunkd print-modinput-config splunkversioncontrol_backup splunkversioncontrol_backup://<your_input_name_goes_here>
If the issue relates to restoration, ensure that the user configured for the restore section has the required access to run the reports that access the _audit index, along with the REST endpoint for users. Finally the user to restore reports must be able to write the knowledge objects.
For further information also refer to the Security Concerns section of this document.
Finally the log files are mentioned under the "Where are the logs?" section of this document
There is also a test file included in the bin directory it can be used with (the path will vary but this assumes you are in $SPLUNK_HOME/etc/apps/):
splunk cmd python SplunkVersionControl/bin/test_git.py -gitRepoURL "git@github..."
To test if the git/SSH setup is working as expected
Both inputs follow a similar validation process:
<srcURL>/servicesNS/nobody/<remoteAppName>/search/jobs/export?search=makeresults (where remoteAppName is SplunkVersionControl unless specified)git ls-remote <gitRepoURL>ssh -n -o \"BatchMode yes\" -o StrictHostKeyChecking=no <gitRepoURL>In 7.3.0 the Splunk process will kill -9 the modular input if it takes more than 30 seconds, if this occurs you can bypass validation by updating the inputs.conf file manually
Yes but do not configure the modular inputs to run on the search head cluster, modular inputs run on each member at the same time which would not work well. What you want to do is configure a standalone server with the modular inputs for backup/restore and set the srcURL/destURL to the remote search head cluster member (or load balanced URL) on the REST port.
This would allow the modular inputs to run backup/restore and any customers to use the dashboard on the search head cluster member to request restoration of a knowledge object
Yes, but there are some tricks involved as I have not yet integrated python git libraries into the application, I'm using the command line!
The first is that the SSH key is still required to access the remote repository, the VersionControl app assumes you can run git clone without a password
In my testing, the SYSTEM users SSH key directory was:
C:\Windows\System32\config\systemprofile\.ssh
Furthermore the id_rsa file within that directory had to have permissions changed on it so only the SYSTEM account had access and no one else...
Furthermore, the git/ssh command may not be in the PATH of the sytem user, and this is fine, you can use the git_command and ssh_command options to point them to your git installation directory, I installed the https://gitforwindows.org/ package into e:\temp\git, I set the two settings to:
E:\temp\git\cmd\git.exe
E:\temp\Git\usr\bin\ssh.exe
Note there is ssh in Windows 10 but it did not work with the same switches as the Unix SSH so I've used the above in testing Furthermore I needed to use a system user command line window which I opened via: psexec –i –s CMD
And set the git global configuration
git config --global user.name "John Doe"
git config --global user.email johndoe@example.com
Finally, I found that the git temporary directory often fails to delete on Windows, since this is done under error conditions it should not be an issue in day to day operation, but I'd recommend using a Linux server if at all possible!
This application, no. But this application can be used to backup a SplunkCloud instance from a remote/on-prem Splunk instance, the same remote instance could also be used to restore to the SplunkCloud instance.
To do this you will need to install Version Control For SplunkCloud on your SplunkCloud instance, and setup this application on a remote/on-prem instance by configuring an interval for both the Splunk Version Control Backup and Splunk Version Control Restore modular inputs
VersionControl For SplunkCloud (stripped down version of this app for SplunkCloud)
SplunkVersionControlCloud github
As of October 2022, there are still no signs of version control within the Splunk Enterprise (or cloud) product, however you do have a few options in terms of a version control app, these include:
Library updates:
Updates:
SplunkVersionControl ChangeDetector Directory savedsearch/services/properties/savedsearches/default (no changes in this version)Library updates:
New features:
Test* will now restore any savedsearch starting with Test, wildcards can be used on any knowledge objecttest_git.pyUpdates:
splunkversioncontrol_restore_class.py_audit index query to look for info=completed as well as info=granted, as this does not appear in Splunk 9_audit log entry to appear, previously it would appear on the same second the dashboard was run, now there is an approx 10 second delayLibrary updates:
Updated README.md Updated Splunk python SDK to 1.6.20
Updated Splunk python SDK to 1.6.18
Corrected an issue where a DELETE combined with other operations could cause a stacktrace if using the run_ko_diff option
Updates to:
splunkversioncontrol_backup_class.py
To correct an issue with recording the git_location of file changes in more cases
Updates to:
splunkversioncontrol_backup_class.py
splunkversioncontrol_restore_class.py
To remove passwords in more cases
Updates to dashboards:
splunkversioncontrol_restore.xml
splunkversioncontrol_restore_dynamic.xml
To provide a drop down list of available knowledge objects in addition to the text field option
Updated reports:
SplunkVersionControl CheckAdmin - simplified to use the Splunk users list
splunk_vc_kom_audit_summary - updated to ignore the manager URI's and handle proxied REST calls from the KOM report
Updated splunk_vc_kom_audit_summary report
Added i=StanzaName to the indexed data when running the audit query
Now attempting to hide (most) passwords from the logs by default (for example when an OS error occurs don't print the stdout including the password in use)
git diff now uses --no-pager to prevent trucation of the diff command with -U0 (no context)
New options:
disable_file_deletion - do not delete files in remote git repo that are not found during backup, useful for testing
use_wdiff - sends the output of the diff command to Unix command wdiff to provide a nicer diff output
Updated report:
SplunkVersionControl ChangeDetector Non-Directory now excludes the CIM Risk and Incident_Management datamodels as they update very frequently with close to zero changes (calculationId changes only)
New option disable_git_ssl_verify
Support for password: syntax for the gitRepoURL parameter when using http/https
Bugfix for proxy code to work with git & HTTP proxies
This version includes a few changes, these include two new parameters on the version control backup:
run_ko_query - if enabled this runs a Splunk savedsearch and adds the additional information of tag=git_tag_name into the output of the modular input which is then indexed
run_ko_diff - if enabled in combination with run_ko_query this additionally adds a diff=git_difference_result from comparing the new version with HEAD~1
To run the query the macro splunk_vc_ko_query, should be configured to point to an appname:searchname, the default is splunk_kom:splunk_vc_kom_audit_summary
If you have the Knowledge Object Overview App for Splunk (https://splunkbase.splunk.com/app/5399/) installed then there is a savedsearch called splunk_vc_kom_audit_summary which can be moved or copied into the splunk_kom app for this new functionality to work as expected
In addition the field qualifiedSearch is now longer backed up for savedsearches
Boolean tickboxes are now used for options that should be true or false
Also attempted to improve the error logging for failed OS process execution
Fixed a few misc bugs related to setting email address/name in the git repo among others
This version includes some changes that should reduce the storage size of savedsearches, in particular:
Note that I have also created https://ideas.splunk.com/ideas/EID-I-1052 as a request to have a way to see the output of savedsearches that is closer to matching the filesystem rather than including 100's of default configuration lines into each savedsearch entry (256 display.visualizations.* attributes per-savedsearch in my prod environment at the time of writing)
This version includes a few major changes:
file_per_ko mode, disabled by default, if enabled outputs 1 file per knowledge object instead of including all knowledge objects of a type within 1 filenext_scheduled_time attribute removed from savedsearches (this results in less unnnessary git commits)If you would like to use file_per_ko this will result in a lot more files in the git repository but this will make it easier to see the history of changes in each file
Note that you must set file_per_ko to true in both the backup & restore for this to work as expected, also if you change the setting you will need to re-create or wipe the repo as the files are stored differently
Updated all dashboards to include version="1.1" tag as required by new Splunk versions
Updated to Splunk python SDK 1.1.16
This version fixes a bug introduced by 1.1.13, version 1.1.13 was removed from SplunkBase due to an error in the code
Updated saved search Splunk Version Control Audit Query POST with new regex
Added the git_branch parameter to allow any branch to be used for backup/restore
Merged pull request from bre77 to make sslVerify option on restore equivalent to the backup version
Fixed sslVerify option to work as expected
Added new parameters into the splunkversioncontrol_backup modular input for:
git_namegit_emailBy default the git global settings will be used, but if specified these will run a git config user.name/git config user.email after cloning the repo README.md updates
Corrected error in splunkversioncontrol_backup.py
README.md update - git repositories must be dedicated per-backup and not shared with other items as the root level / top level directory is used Merged pull request from calesanz https://github.com/gjanders/SplunkVersionControl/pull/16 to allow a new sslVerify option to pass in the CA certificate file, or to leave SSL validation disabled In addition this pull request adds a requestingAddress which optionally controls the call-back ip when using the postversioncontrolrestore command Finally this pull requests adds scripts and a testing suite using docker into the github version, for SplunkBase the test directory is removed (you can access it on https://github.com/gjanders/SplunkVersionControl)
Updated Splunk python SDK to 1.6.15
Increase timeout for commands to a default of 60 seconds Ensure a valid message is sent back to the user if a dynamic restore fails If git checkout times out, cancel the restore attempt
Allow the backup process to run on search head clusters for those that wish to do this...
Minor update so that the gitTempDir refers to the correct directory and not a sub-directory
Created inputs.conf to pass app inspect and force python 3 by default
password: syntax did not work if using a dynamic/REST based restore, now supported
Found a bug that stops this running on Splunk 8 / python 3
Corrected useLocalAuth setting so that it works as expected
Corrected imports so that post version control method works as well as the cloud version
Now tested on Windows and Splunk Cloud (note this version of the app is not installed on SplunkCloud, the VersionControl for SplunkCloud is the app to install on the SplunkCloud instance, this variation of the app includes only what is required to remotely backup/restore a SplunkCloud instance
This app is still used for SplunkCloud instances, but this app is installed on-prem
Updates include:
The README.md has had various updates including more details around setup and how this was tested on Windows
Fixed missing sys import from splunkversioncontrol_rest_restore.py
Updated README.md instructions
Updated python SDK to version 1.6.12
Updated inputs.conf.spec and restmap.conf to specify python3 as the default version to pass appinspect
Corrected errors in the import of the six library which stopped this from working Minor updates to README.md
Changed import to use local Splunk python SDK to ensure this works on older Splunk versions Added the (experimental) apps list option to attempt to make this work with Splunk Cloud instances
Version 1.0.8 caused an issue where the checkpoint file stopped getting created, fixed in this version
1.0.8 has Splunk 8 / python 3 changes only
This version has a few major changes:
splunkversioncontrol_restore modular input must still exist, it is not required to run on a scheduleThe new dashboard splunkversioncontrol_restore_dynamic which is now the default dashboard is an alternative to the splunkversioncontrol_restore dashboard which remains lookup based (the latter dashboard assumes the splunkversioncontrol_restore modular input is running on a schedule
Note that if you are running this app on a search head cluster, and restoring from a different server you may wish to remove the:
web.confrestmap.confFiles from the default directory, this removes the ability to trigger a restore by hitting a REST endpoint without authentication
Dashboard backups no longer include version attribute (appears on some dashboards and prevents restoration) Updated README.md to include an installation and troubleshooting guide
Correct lastRunEpoch (as per fix in 1.0.3) for macros
Minor changes to the code to wipe the git directory re-clone on failure in both the clone failure & checkout master / git pull scenarios
Fix so that the first run does not fail due to the lastRunEpoch been set to None
Minor log improvement for when the saved searches do not work as expected
Minor tweak to logging to ensure timezones are handled when logging updated objects
Minor changes to log when an object has an updated timestamp newer than the last epoch value (this logs objects that have a newer timestamp since last run)
Improvements to logging for git related errors and auto-wipe of the git repo on failure (this handles corruption of git repos on disk)
Change of app icons only, no functional changes
Added the sort_keys option into the python code, this should ensure the output files for git are in a consistent order (previously random).
The goal is to reduce the git repository size increase over time
Added Troubleshooting section in details/README.md about "OPENSSL not found" issues on Ubuntu