ConnectionResetError: [Errno 104] Connection reset by peer

[DaniloOliveira28]

Bug Description

We are using the cloudant-python lib to handle our connections with our database. Randomly the code raises a ConnectionResetError.

1. Steps to reproduce and the simplest code sample possible to demonstrate the issue

I could not create a external scenario to reproduce it, but I will outline the context of the error scenario.

I have an code that make iterative calls to our couchdb. Every loop make the same call to an specific view, but with different keys.

Sometimes, the request is not sent and the program return the error below.

At the first I thought that could be something related with our proxy - Haproxy. But, I have tested the script connecting directly with the database and the error continued. On the next, without the HAPROXY, I replaced the request call (from the python-cloudant) to a simple request (from request package) and the error disappeared, so I think that the error is related with this lib. In addition, my couchdb server does not log any request call, so, its is one more info that suggest that the error is on the client side.

2. What you expected to happen

The script should make the requests and finish without any error.

3. What actually happened

Eventually in some requests the script raise the error below.

2021-02-10 11:16:14 ERROR: Exception occurred on parse_candidate
Traceback (most recent call last):
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/urllib3/util/retry.py", line 531, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/common/decorators/log_exception.py", line 8, in _log_exception_and_ignore
    return function(*args, **kwargs)
  File "/home/danilo/Workspace/stakeholders-extraction/sl_stakeholders_extraction/tse/parsers/candidature_parser.py", line 57, in parse_candidate
    candidate_info['CurrentParty'] = get_party(candidature['SiglaPartido'])
  File "/home/danilo/Workspace/stakeholders-extraction/sl_stakeholders_extraction/tse/parsers/__init__.py", line 5, in get_party
    return get_party_by_initials(party_initials)
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/common/utils/parties.py", line 16, in get_party_by_initials
    return _get_party(initials, PARTIES_VIEW_NAME, 'by_initials')
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/common/utils/parties.py", line 21, in _get_party
    party = _get_party_from_view(ddoc_id, view_name, key)
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/common/utils/parties.py", line 28, in _get_party_from_view
    view_result = PARTIES_DB.get_view_result(view_name=view_name,
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/cloudant/result.py", line 427, in all
    return self[:]
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/cloudant/result.py", line 223, in __getitem__
    data = self._ref(**self.options)
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/cloudant/view.py", line 236, in __call__
    resp = get_docs(self._r_session,
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/cloudant/_common_util.py", line 261, in get_docs
    resp = r_session.get(url, headers=headers, params=f_params)
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/cloudant/_client_session.py", line 130, in request
    return super(BasicSession, self).request(
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/cloudant/_client_session.py", line 65, in request
    resp = super(ClientSession, self).request(
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/danilo/Workspace/stakeholders-extraction/env/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Environment details

• Cloudant

  2.14.0

• Python version

  3.9.1

[mojito317]

Hi @DaniloOliveira28!

Have you tried to put a time.sleep(0.01) before your request? This answer says that sometimes these types of errors can be corrected with a time.sleep(0.01).

[DaniloOliveira28]

I have tried to put time.sleep(0.01) before the get_view_result method, but it does not work. The error still happening.

[DaniloOliveira28]

News about the error.

The error only happens if there is a big time gap between the calls to couchdb. My scenario, in each iteration I call couchdb to get a specific doc. To avoid make many calls, I did a local cache (a dict with the keys and the response). I removed this cache, so now, every time couchdb is being called. In this scenario, the error did not happen.

[DaniloOliveira28]

In my case, the gap between the requests is ~ 410s

[mojito317]

The error only happens if there is a big time gap between the calls to couchdb. My scenario, in each iteration I call couchdb to get a specific doc. To avoid make many calls, I did a local cache (a dict with the keys and the response). I removed this cache, so now, every time couchdb is being called. In this scenario, the error did not happen.

If you are leaving connections idle longer than the server will keep them alive you need to disable HTTP keep-alive or handle the error and then retry the request.

You can probably disable the HTTP keep-alive by client.r_session.keep_alive = False, or with the Connection: close request header if requests does not overwrite that.

Another option would be to configure your server/proxy to keep connections alive for longer.

[DaniloOliveira28]

How can I know if I am leaving connections idle? I though that the python-cloudant or request would manage these kind of behaviours automatically.

[DaniloOliveira28]

@mojito317 I could fix it!

The error was happening because the server was killing our connection. To avoid it, we need to add the following header on the client.py, as you suggested.

What do you think of a PR to add a param on CouchDB class to allow the user decide if he want to put the header or not?

[mojito317]

What do you think of a PR to add a param on CouchDB class to allow the user decide if he want to put the header or not?

I think there is no need to do that, since the r_session is already exposed to the user:

import os

from cloudant.client import Cloudant
client = Cloudant(os.environ['SERVER_USERNAME'], os.environ['SERVER_PASSWORD'], url=os.environ['SERVER_URL'], connect=True)

# you can set this way:
client.r_session.headers.update({'Connection': 'close'})

Edit: I double-checked it, and the client.r_session.keep_alive option did not work (maybe it was a feature of requests 1.x) but setting the headers does like in the example above.

[DaniloOliveira28]

Ty @mojito317 .