After deploying my Flask app successfully several times, zappa deploy stopped working. When I try to deploy, I get the following error:
Read timeout on endpoint URL: "{not disclosing my URL for security reasons}"
Error: Unable to upload to S3. Quitting.
Expected Behavior
When deploying, Zappa should create (and has created using this machine in the past) all necessary resources and upload my zipped project to S3.
Actual Behavior
The app package is created and zipped, then the IAM policy and S3 bucket are created successfully. Then I see a progress bar (which does not update regularly, but only a handful times over the course of several minutes). After ~5 minutes or so, the progress bar disappears. Then, after another ~5 minutes, I get the error message above.
I have replaced the statement that prints this exception with a call to logging.exception, which gave the following stacktrace:
13:19 $ zappa deploy
Calling deploy for stage dev..
Downloading and installing dependencies..
- psycopg2-binary==2.8.4: Using locally cached manylinux wheel
- markupsafe==1.1.1: Using locally cached manylinux wheel
- sqlite==python3: Using precompiled lambda package
'python3.7'
Packaging project as zip.
Uploading fabian-test-zappa-poging-12006-dev-1573215672.zip (23.4MiB)..
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24.5M/24.5M [05:40<00:00, 20.4KB/s]ERROR:Read timeout on endpoint URL: "https://fabian-test-zappa-12006.s3.eu-west-1.amazonaws.com/fabian-test-zappa-poging-12006-dev-1573215672.zip?uploadId=u6kP9nuJRwK0wFBFOVa9kIJECBYftFW3p1a0o__Ne9SxKT4i7jmvY_0rwJ4uy7Zapo0JzAFRgrn9nwJN4RothWiO9_7G_MXhQUAistjbJ9QVmWBX0ZnZUEGB572T2jjH&partNumber=1"
Traceback (most recent call last):
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1321, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 296, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.7/ssl.py", line 911, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='fabian-test-zappa-12006.s3.eu-west-1.amazonaws.com', port=443): Read timed out. (read timeout=60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/zappa/core.py", line 962, in upload_to_s3
Callback=progress.update
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 131, in upload_file
extra_args=ExtraArgs, callback=Callback)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/boto3/s3/transfer.py", line 279, in upload_file
future.result()
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 265, in result
raise self._exception
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/upload.py", line 722, in _main
Body=body, **extra_args)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
operation_model, request_dict, request_context)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 667, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
return self.http_session.send(request)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://fabian-test-zappa-12006.s3.eu-west-1.amazonaws.com/fabian-test-zappa-poging-12006-dev-1573215672.zip?uploadId=W5Ishdue_X_UFTLhPVDQ.TCR600JN1GxNHEGE9RkRaY.fWWElxjvSDQ2IOiwH.A4eg7fCYBUNjBdWikVY9Mz3nPtIBSgK3MkShShiMRtpcKfC2uW_jniCCblTJsMkKaE&partNumber=2"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/zappa/core.py", line 965, in upload_to_s3
self.s3_client.upload_file(source_path, bucket_name, dest_path)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 131, in upload_file
extra_args=ExtraArgs, callback=Callback)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/boto3/s3/transfer.py", line 279, in upload_file
future.result()
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/futures.py", line 265, in result
raise self._exception
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/s3transfer/upload.py", line 722, in _main
Body=body, **extra_args)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
operation_model, request_dict, request_context)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/client.py", line 667, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
return self.http_session.send(request)
File "/c/Users/Fabian.vanDijk/Documents/zelfstudie/zappa_test2/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "{not disclosing the URL for obvious reasons}"
Error: Unable to upload to S3. Quitting.
Possible Fix
I know no possible fix, but I have tested the following:
My teammates can zappa deploy with the exact code/configuration that I use, using their machines.
They can also use zappa deploy using an aws profile with an access key to my account, which means this is probably not a permissions issue.
I have deployed to a different AZ, which gives the same error.
I have thrown away my repo and virtualenv and made a new one. The error persists.
I have successfully uploaded a file to this S3 bucket with boto3.
Your Environment
Zappa version used: 0.48.2
Operating System and Python version: Ubuntu 18.04, Python 3.7.5
After deploying my Flask app successfully several times,
zappa deploy
stopped working. When I try to deploy, I get the following error:Expected Behavior
When deploying, Zappa should create (and has created using this machine in the past) all necessary resources and upload my zipped project to S3.
Actual Behavior
The app package is created and zipped, then the IAM policy and S3 bucket are created successfully. Then I see a progress bar (which does not update regularly, but only a handful times over the course of several minutes). After ~5 minutes or so, the progress bar disappears. Then, after another ~5 minutes, I get the error message above.
I have replaced the statement that
print
s this exception with a call tologging.exception
, which gave the following stacktrace:Possible Fix
I know no possible fix, but I have tested the following: My teammates can
zappa deploy
with the exact code/configuration that I use, using their machines. They can also usezappa deploy
using an aws profile with an access key to my account, which means this is probably not a permissions issue. I have deployed to a different AZ, which gives the same error. I have thrown away my repo and virtualenv and made a new one. The error persists. I have successfully uploaded a file to this S3 bucket with boto3.Your Environment
0.48.2
pip freeze
:zappa_settings.py
: