ReadTimeoutError on Zappa Deploy

After deploying my Flask app successfully several times, zappa deploy stopped working. When I try to deploy, I get the following error:

Read timeout on endpoint URL: "{not disclosing my URL for security reasons}"

Error: Unable to upload to S3. Quitting.

Expected Behavior

When deploying, Zappa should create (and has created using this machine in the past) all necessary resources and upload my zipped project to S3.

Actual Behavior

The app package is created and zipped, then the IAM policy and S3 bucket are created successfully. Then I see a progress bar (which does not update regularly, but only a handful times over the course of several minutes). After ~5 minutes or so, the progress bar disappears. Then, after another ~5 minutes, I get the error message above.

I have replaced the statement that prints this exception with a call to logging.exception, which gave the following stacktrace:

13:19 $ zappa deploy
Calling deploy for stage dev..
Downloading and installing dependencies..
 - psycopg2-binary==2.8.4: Using locally cached manylinux wheel
 - markupsafe==1.1.1: Using locally cached manylinux wheel
 - sqlite==python3: Using precompiled lambda package
'python3.7'
Packaging project as zip.
Uploading fabian-test-zappa-poging-12006-dev-1573215672.zip (23.4MiB)..
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24.5M/24.5M [05:40<00:00, 20.4KB/s]ERROR:Read timeout on endpoint URL: "https://fabian-test-zappa-12006.s3.eu-west-1.amazonaws.com/fabian-test-zappa-poging-12006-dev-1573215672.zip?uploadId=u6kP9nuJRwK0wFBFOVa9kIJECBYftFW3p1a0o__Ne9SxKT4i7jmvY_0rwJ4uy7Zapo0JzAFRgrn9nwJN4RothWiO9_7G_MXhQUAistjbJ9QVmWBX0ZnZUEGB572T2jjH&partNumber=1"
Traceback (most recent call last):
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.7/ssl.py", line 911, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='fabian-test-zappa-12006.s3.eu-west-1.amazonaws.com', port=443): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/zappa/core.py", line 962, in upload_to_s3
    Callback=progress.update
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 131, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/boto3/s3/transfer.py", line 279, in upload_file
    future.result()
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/upload.py", line 722, in _main
    Body=body, **extra_args)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
    success_response, exception):
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
    return self.http_session.send(request)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://fabian-test-zappa-12006.s3.eu-west-1.amazonaws.com/fabian-test-zappa-poging-12006-dev-1573215672.zip?uploadId=W5Ishdue_X_UFTLhPVDQ.TCR600JN1GxNHEGE9RkRaY.fWWElxjvSDQ2IOiwH.A4eg7fCYBUNjBdWikVY9Mz3nPtIBSgK3MkShShiMRtpcKfC2uW_jniCCblTJsMkKaE&partNumber=2"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/zappa/core.py", line 965, in upload_to_s3
    self.s3_client.upload_file(source_path, bucket_name, dest_path)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 131, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/boto3/s3/transfer.py", line 279, in upload_file
    future.result()
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/upload.py", line 722, in _main
    Body=body, **extra_args)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
    success_response, exception):
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
    return self.http_session.send(request)
  File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "{not disclosing the URL for obvious reasons}"
Error: Unable to upload to S3. Quitting.

Possible Fix

I know no possible fix, but I have tested the following: My teammates can zappa deploy with the exact code/configuration that I use, using their machines. They can also use zappa deploy using an aws profile with an access key to my account, which means this is probably not a permissions issue. I have deployed to a different AZ, which gives the same error. I have thrown away my repo and virtualenv and made a new one. The error persists. I have successfully uploaded a file to this S3 bucket with boto3.

Your Environment

Zappa version used: 0.48.2
Operating System and Python version: Ubuntu 18.04, Python 3.7.5

The output of pip freeze:

apispec==1.3.3
argcomplete==1.9.3
attrs==19.3.0
Babel==2.7.0
boto3==1.10.12
botocore==1.13.12
certifi==2019.9.11
cfn-flip==1.2.2
chardet==3.0.4
Click==7.0
colorama==0.4.1
defusedxml==0.6.0
docutils==0.15.2
durationpy==0.5
Flask==1.1.1
Flask-AppBuilder==2.2.0
Flask-Babel==0.12.2
Flask-JWT-Extended==3.24.1
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.1
Flask-WTF==0.14.2
future==0.16.0
hjson==3.0.1
idna==2.8
importlib-metadata==0.23
itsdangerous==1.1.0
Jinja2==2.10.3
jmespath==0.9.3
jsonschema==3.1.1
kappa==0.6.0
lambda-packages==0.20.0
MarkupSafe==1.1.1
marshmallow==2.19.5
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.19.0
more-itertools==7.2.0
pipdeptree==0.13.2
placebo==0.9.0
prison==0.1.2
psycopg2-binary==2.8.4
PyJWT==1.7.1
pyrsistent==0.15.5
python-dateutil==2.6.1
python-slugify==1.2.4
python3-openid==3.1.0
pytz==2019.3
PyYAML==5.1.2
requests==2.22.0
s3transfer==0.2.1
six==1.13.0
SQLAlchemy==1.3.10
SQLAlchemy-Utils==0.35.0
toml==0.10.0
tqdm==4.19.1
troposphere==2.5.2
Unidecode==1.1.1
urllib3==1.25.6
Werkzeug==0.16.0
wsgi-request-logger==0.4.6
WTForms==2.2.1
zappa==0.48.2
zipp==0.6.0

Link to your project (optional):

Your zappa_settings.py:

{
"dev": {
    "app_function": "app.app_init.app",
    "aws_region": "eu-west-1",
    "profile_name": "default",
    "project_name": "fabian-test-zappa-poging-12006",
    "runtime": "python3.7",
    "s3_bucket": "fabian-test-zappa-12006"
}
}

Miserlou / Zappa