search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
9.01k stars 1.87k forks source link

Boto3 raises ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) #1072

Closed godlark closed 7 years ago

godlark commented 7 years ago

I have an application which makes file uploading to s3. The app works from local machine but doesn't work from my server. On the server boto3 raises:

[2017-04-27 17:39:53,595: DEBUG/PoolWorker-1] Rewinding stream: <_io.BytesIO object at 0x7f33c3d31a40>
[2017-04-27 17:39:53,595: DEBUG/PoolWorker-1] Event request-created.s3.UploadPart: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f33c6c2c1d0>>
[2017-04-27 17:39:53,595: DEBUG/PoolWorker-1] Event choose-signer.s3.UploadPart: calling handler <function set_operation_specific_signer at 0x7f33c6e6f620>
[2017-04-27 17:39:53,596: DEBUG/PoolWorker-1] Calculating signature using hmacv1 auth.
[2017-04-27 17:39:53,596: DEBUG/PoolWorker-1] HTTP request method: PUT
[2017-04-27 17:39:53,596: DEBUG/PoolWorker-1] StringToSign:
PUT
eR0MufFLtcWDshaxhn8IUQ==

Thu, 27 Apr 2017 17:39:53 GMT
/file-upload-test/Fedora-Workstation-netinst-x86_64-25-1.3.iso?partNumber=2&uploadId=PDI6OPSGWC4R6NBHX1QAWF72EF41TH0Z1XSJ6UYYGUETVYCZOESQFCFW8ZMXGOLI
[2017-04-27 17:39:53,596: DEBUG/PoolWorker-1] Sending http request: <PreparedRequest [PUT]>
[2017-04-27 17:39:53,596: INFO/PoolWorker-1] Starting new HTTP connection (4): objectstorage.esclor86.eecloud.nsn-rdnet.net
[2017-04-27 17:39:53,599: DEBUG/PoolWorker-1] Waiting for 100 Continue response.
[2017-04-27 17:39:53,738: INFO/MainProcess] Received task: tasks.upload_chunk[f99a70a5-a57a-46fa-9b76-3fdaa4fce4cb]  
[2017-04-27 17:39:54,600: DEBUG/PoolWorker-1] No response seen from server, continuing to send the response body.
10.154.117.101 - - [27/Apr/2017 17:39:55] "GET /upload_progress?flowIdentifier=486539264-Fedora-Workstation-netinst-x86_64-25-13iso HTTP/1.1" 200 -
[2017-04-27 17:39:55,644: DEBUG/PoolWorker-1] ConnectionError received when sending HTTP request.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python3.5/dist-packages/botocore/awsrequest.py", line 130, in _send_request
    self, method, url, body, headers, *args, **kwargs)
  File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python3.5/dist-packages/botocore/awsrequest.py", line 182, in _send_output
    self.send(message_body)
  File "/usr/local/lib/python3.5/dist-packages/botocore/awsrequest.py", line 242, in send
    return HTTPConnection.send(self, str)
  File "/usr/lib/python3.5/http/client.py", line 905, in send
    self.sock.sendall(datablock)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/packages/urllib3/util/retry.py", line 245, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/packages/urllib3/packages/six.py", line 309, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python3.5/dist-packages/botocore/awsrequest.py", line 130, in _send_request
    self, method, url, body, headers, *args, **kwargs)
  File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python3.5/dist-packages/botocore/awsrequest.py", line 182, in _send_output
    self.send(message_body)
  File "/usr/local/lib/python3.5/dist-packages/botocore/awsrequest.py", line 242, in send
    return HTTPConnection.send(self, str)
  File "/usr/lib/python3.5/http/client.py", line 905, in send
    self.sock.sendall(datablock)
botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/botocore/endpoint.py", line 204, in _get_response
    proxies=self.proxies, timeout=self.timeout)
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Version of boto3: 1.4.4 Version of botocore: 1.5.44 Version of python: 3.5.3

Creating multipart upload works on both machines, only upload_part doesn't work. The interesting fragment of code:

def upload_part(s3, key, part_number, bucket, body, upload_id):
    i = 0
    while True:
        try:
            print("Sleeping")
            time.sleep(2 ** i * randint(0, 16))
            r = s3.upload_part(Key=key, PartNumber=part_number, Bucket=bucket, UploadId=upload_id, Body=body)
            break
        except ReadTimeout as e:
            i += 1
            if i > 10:
                raise e
        except botocore.vendored.requests.exceptions.ConnectionError as e:
            i += 1
            if i > 10:
                raise e
        except BaseException as e:
            print(repr(e))
            raise e
    print("Uploaded part %d" % part_number)
    return len(body), r["ETag"]

The function 'upload_part' is called by celery task, but I've decreased concurrency to 1.

dstufft commented 7 years ago

This error is originating from the network, somewhere in the network path something is resetting the TCP connection and causing that error. Since this only occurs on your server, it's likely something amiss with the network that the server itself is connected to.