splunk / splunk-app-examples

App examples for Splunk Enterprise
Apache License 2.0
119 stars 72 forks source link

Intermittent http.client.RemoteDisconnected error #41

Closed NilSerra closed 1 year ago

NilSerra commented 1 year ago

I'm getting intermittent http.client.RemoteDisconnected errors when performing queries to Splunk. After successfully creating a Splunk Service instance, i create a normal search job:

def splunk_search(service, search_query, kwargs_search):
    job_normalsearch = service.jobs.create(search_query, **kwargs_search)
    print("- Starting Splunk search")

    while True:
        while not job_normalsearch.is_ready():
            pass
        stats = {"isDone": job_normalsearch["isDone"],
                "doneProgress": float(job_normalsearch["doneProgress"])*100,
                "scanCount": int(job_normalsearch["scanCount"]),
                "eventCount": int(job_normalsearch["eventCount"]),
                "resultCount": int(job_normalsearch["resultCount"])}

        status = ("\r%(doneProgress)03.1f%%   %(scanCount)d scanned   "
                "%(eventCount)d matched   %(resultCount)d results") % stats

        sys.stdout.write(status)
        sys.stdout.flush()
        if stats["isDone"] == "1":
            sys.stdout.write("\n- Done!\n")
            break
        sleep(0.5)

    query_result = job_normalsearch.results(count=0, output_mode='json')
    job_normalsearch.cancel()

    return query_result

This function is sometimes successful, but for other runs I'm receiving the following error:

File "test.py", line 19, in splunk_search
    while not job_normalsearch.is_ready():
  File "C:\Users\user\.conda\envs\testenv\lib\site-packages\splunklib\client.py", line 2862, in is_ready
    response = self.get()
  File "C:\Users\user\.conda\envs\testenv\lib\site-packages\splunklib\client.py", line 1117, in get
    return super(Entity, self).get(path_segment, owner=owner, app=app, sharing=sharing, **query)
  File "C:\Users\user\.conda\envs\testenv\lib\site-packages\splunklib\client.py", line 862, in get
    return self.service.get(path,
  File "C:\Users\user\.conda\envs\testenv\lib\site-packages\splunklib\binding.py", line 289, in wrapper
    return request_fun(self, *args, **kwargs)
  File "C:\Users\user\.conda\envs\testenv\lib\site-packages\splunklib\binding.py", line 73, in new_f
    val = f(*args, **kwargs)
  File "C:\Users\user\.conda\envs\testenv\lib\site-packages\splunklib\binding.py", line 699, in get
    response = self.http.get(path, all_headers, **query)
  File "C:\Users\user\.conda\envs\testenv\lib\site-packages\splunklib\binding.py", line 1232, in get
    return self.request(url, { 'method': "GET", 'headers': headers })
  File "C:\Users\user\.conda\envs\testenv\lib\site-packages\splunklib\binding.py", line 1294, in request
    response = self.handler(url, message, **kwargs)
  File "C:\Users\user\.conda\envs\testenv\lib\site-packages\splunklib\binding.py", line 1453, in request
    response = connection.getresponse()
  File "C:\Users\user\.conda\envs\testenv\lib\http\client.py", line 1374, in getresponse
    response.begin()
  File "C:\Users\user\.conda\envs\testenv\lib\http\client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "C:\Users\user\.conda\envs\testenv\lib\http\client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

I have also tried using one_shot searches and also getting the same error intermittently in the following line:

service.jobs.oneshot(search_query, **kwargs_search)

Do you know what could be causing this problem?

NilSerra commented 1 year ago

Looks like I have solved the problem. Changing the token from parameter from "splunkToken" to "token" when creating the service has solved the problem.