I have a huge (millions of records) kvstore that I am trying to prune based off age. This usually works fine with a few hundred at a time, but when I run with 100000 or higher rows at a time, it will cause the a spike in the number of time wait sockets on the Splunk box, and in turn causes occasional UI issues elsewhere in Splunk since it can't open a new socket.
Looking at search.log for the search running deletekeys, it has a bunch of errors that look to be similar problems where it can create a new socket.
11-30-2021 13:02:04.779 ERROR ScriptRunner [40268 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/kvstore_tools/bin/kvstore_deletekeys.py __EXECUTE__ collection=event_history': OSError: [Errno 99] Cannot assign requested address
11-30-2021 13:02:04.779 ERROR ScriptRunner [40268 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/kvstore_tools/bin/kvstore_deletekeys.py __EXECUTE__ collection=event_history': During handling of the above exception, another exception occurred:
11-30-2021 13:02:04.779 ERROR ScriptRunner [40268 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/kvstore_tools/bin/kvstore_deletekeys.py __EXECUTE__ collection=event_history': Traceback (most recent call last):
I have a huge (millions of records) kvstore that I am trying to prune based off age. This usually works fine with a few hundred at a time, but when I run with 100000 or higher rows at a time, it will cause the a spike in the number of time wait sockets on the Splunk box, and in turn causes occasional UI issues elsewhere in Splunk since it can't open a new socket.
Looking at search.log for the search running
deletekeys
, it has a bunch of errors that look to be similar problems where it can create a new socket.Glancing at the code, it is using urllib, but Splunk's python does come with Requests, why not use Requests and urllib3, where we can reuse existing ones already used? https://docs.python-requests.org/en/latest/user/advanced/#keep-alive
Also, too bad Splunk's KVStore REST api does not let you pass in multiple keys to delete at a time...
Granted, this is only for cases where you are deleting a bunch of items from the kvstore, but it might make sense in other cases.