Add optional retries to connection attempts

[mew1033]

I have a long running process that connects to Splunk and executes and manages multiple jobs over (potentially) hours. Every now and then I'll get a Connection reset by peer error that blows everything up. This should allow me to build my Splunk connection object with a little retry logic built into the sdk.

Traceback I'm getting for reference:

Traceback (most recent call last):
  File "/opt/myCode/myCode.py", line 2130, in <module>
    main()
  File "/opt/myCode/myCode.py", line 1308, in main
    config_data['fields']['matchFields'])
  File "/opt/myCode/myCode.py", line 839, in rebuild_lookups_from_database
    while not all([z.is_done() for z in jobs]):
  File "/usr/lib/python2.7/site-packages/splunklib/client.py", line 2703, in is_done
    if not self.is_ready():
  File "/usr/lib/python2.7/site-packages/splunklib/client.py", line 2715, in is_ready
    response = self.get()
  File "/usr/lib/python2.7/site-packages/splunklib/client.py", line 1009, in get
    return super(Entity, self).get(path_segment, owner=owner, app=app, sharing=sharing, **query)
  File "/usr/lib/python2.7/site-packages/splunklib/client.py", line 766, in get
    **query)
  File "/usr/lib/python2.7/site-packages/splunklib/binding.py", line 290, in wrapper
    return request_fun(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/splunklib/binding.py", line 71, in new_f
    val = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/splunklib/binding.py", line 680, in get
    response = self.http.get(path, all_headers, **query)
  File "/usr/lib/python2.7/site-packages/splunklib/binding.py", line 1184, in get
    return self.request(url, { 'method': "GET", 'headers': headers })
  File "/usr/lib/python2.7/site-packages/splunklib/binding.py", line 1242, in request
    response = self.handler(url, message, **kwargs)
  File "/usr/lib/python2.7/site-packages/splunklib/binding.py", line 1386, in request
    response = connection.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib64/python2.7/ssl.py", line 757, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 651, in read
    v = self._sslobj.read(len or 1024)
error: [Errno 104] Connection reset by peer```

[mew1033]

I've been running with these changes on my copy of the sdk for the past 3 weeks and it has completely solved my connection reset problem. I'm not sure what else should be tested, but this is working and working well for me.

[mew1033]

Any comments here? This has definitely helped with my processes.

[shakeelmohamed]

Hi @mew1033, we unfortunately have not had a chance to look at this PR due to resource constraints. I will personally see if we can get this merged in sooner rather than later. Thank you for your patience and understanding.

[mew1033]

@shakeelmohamed Sounds good. Thank you for your response. I'll keep using my patched version for now.

[mew1033]

@shakeelmohamed Just wanted to follow up on this PR. Any chance of getting it merged? I Just rebased to current develop.

[ashah-splunk]

Hi @mew1033 thank you for your PR. We have had a look at the changes and just wanted to check with you the reason behind keeping the default retry time to 10s. Also would request to resolve the conflicts as well as see if you can add a unit test for the changes. Thank you for your patience and understanding.