googleapis / python-storage

Apache License 2.0
440 stars 152 forks source link

Larger uploads time out with `requests.exceptions.ConnectionError` #263

Closed kylefoley76 closed 4 years ago

kylefoley76 commented 4 years ago

I'm getting this error every time for files which are really not all that big. I can upload files which are 2 megs, but I cannot upload files larger than 7 megs. I have gotten this bug a lot in the past and I used to think it depended on my slow internet connection upload speed. So in the past 3 megs per second was too slow. But now it will not even work with an upload connection speed of 10 megs. Still, I think Google should be capable of uploading large files even with a slow connection speed.

Python version: 3.8.0 Google-Cloud-Version: 1.31.0

Traceback (most recent call last):
  File "/Users/kylefoley/codes/pcode/other/text2audio.py", line 284, in <module>
    audio2txt()
  File "/Users/kylefoley/codes/pcode/other/text2audio.py", line 22, in __init__
    self.loop_audio2txt()
  File "/Users/kylefoley/codes/pcode/other/text2audio.py", line 54, in loop_audio2txt
    self.upload2cloud()
  File "/Users/kylefoley/codes/pcode/other/text2audio.py", line 67, in upload2cloud
    bucket.blob(f'audio/{self.file}').upload_from_string(self.content)
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 2437, in upload_from_string
    self.upload_from_file(
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 2220, in upload_from_file
    created_json = self._do_upload(
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 2067, in _do_upload
    response = self._do_resumable_upload(
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1946, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport, timeout=timeout)
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/resumable_media/requests/upload.py", line 495, in transmit_next_chunk
    response = _request_helpers.http_request(
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/resumable_media/requests/_request_helpers.py", line 136, in http_request
    return _helpers.wait_and_retry(func, RequestsMixin._get_status_code, retry_strategy)
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/resumable_media/_helpers.py", line 165, in wait_and_retry
    response = func()
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/auth/transport/requests.py", line 464, in request
    response = super(AuthorizedSession, self).request(
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out'))

Here is a snippet of the code I'm using. Again, this code works with small audio files.

        storage_client = storage.Client()
        bucket_name = f'deduction4'
        bucket = storage_client.get_bucket(bucket_name)
        bucket.blob(f'audio/{self.file}').upload_from_string(self.content)
tseaver commented 4 years ago

Your request is taking longer than the default timeout (60 seconds): you can pass a longer value, e.g.:

        storage_client = storage.Client()
        bucket_name = f'deduction4'
        bucket = storage_client.get_bucket(bucket_name)
        bucket.blob(f'audio/{self.file}').upload_from_string(
            self.content, timeout=240
        )