"SSL validation failed" when using async @task

johannesjasper commented 3 years ago

When using async tasks with Zappa, I get botocore.exceptions.SSLError: SSL validation failed for <AWS domain>[Errno 2] No such file or directory

Context

I am deploying a Falcon (WSGI) app with an endpoint that triggers a long-running task. In order to avoid request timeouts, I am using the @task decorator around the long-running function.

Example

See this minimal example.

Expected Behavior

Using the @task decorator should work

Actual Behavior

As soon as I use the @task decorator, my requests fail with the following stack trace

Traceback (most recent call last):
  File "/var/task/botocore/httpsession.py", line 254, in send
  urllib_response = conn.urlopen(
  File "/var/task/urllib3/connectionpool.py", line 719, in urlopen
  retries = retries.increment(
  File "/var/task/urllib3/util/retry.py", line 376, in increment
  raise six.reraise(type(error), error, _stacktrace)
  File "/var/task/six.py", line 703, in reraise
  raise value
  File "/var/task/urllib3/connectionpool.py", line 665, in urlopen
  httplib_response = self._make_request(
  File "/var/task/urllib3/connectionpool.py", line 376, in _make_request
  self._validate_conn(conn)
  File "/var/task/urllib3/connectionpool.py", line 996, in _validate_conn
  conn.connect()
  File "/var/task/urllib3/connection.py", line 352, in connect
  self.sock = ssl_wrap_socket(
  File "/var/task/urllib3/util/ssl_.py", line 338, in ssl_wrap_socket
  raise SSLError(e)
urllib3.exceptions.SSLError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/botocore/endpoint.py", line 200, in _do_get_response
  http_response = self._send(request)
  File "/var/task/botocore/endpoint.py", line 269, in _send
  return self.http_session.send(request)
  File "/var/task/botocore/httpsession.py", line 281, in send
  raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://lambda.eu-central-1.amazonaws.com/2015-03-31/functions/xxx/invocations [Errno 2] No such file or directory

When I use @task_sns I get the same result, except with

botocore.exceptions.SSLError: SSL validation failed for https://sts.amazonaws.com/ [Errno 2] No such file or directory

It seems to me that within the lambda running the WSGI app it tries to invoke itself (the same lambda) but cannot validate the SSL certificates of the AWS API.

Your Environment

Zappa version used: 0.51.0
Operating System and Python version: Ubuntu 20.04

The output of pip freeze:

argcomplete==1.12.0
boto3==1.14.45
botocore==1.17.45
certifi==2020.6.20
cfn-flip==1.2.3
chardet==3.0.4
click==7.1.2
docutils==0.15.2
durationpy==0.5
falcon==2.0.0
future==0.18.2
hjson==3.0.1
idna==2.10
jmespath==0.10.0
kappa==0.6.0
pip-tools==5.3.1
placebo==0.9.0
pydantic==1.6.1
python-dateutil==2.6.1
python-slugify==4.0.1
PyYAML==5.3.1
requests==2.24.0
s3transfer==0.3.3
six==1.15.0
text-unidecode==1.3
toml==0.10.1
tqdm==4.48.2
troposphere==2.6.2
urllib3==1.25.10
Werkzeug==0.16.1
wsgi-request-logger==0.4.6
zappa==0.51.0

Your zappa_settings.json:

{
"dev": {
"aws_region": "eu-central-1",
"app_function": "server.falcon_app",
"profile_name": "default",
"project_name": "zappa_error_demo",
"runtime": "python3.8",
"s3_bucket": "zappa-error-demo",
"memory_size": 128,
"exclude": [
  "*.zip",
  ".idea",
  "local_config.py",
  ".pytest_cache"
]
}
}

Things I read and tried

I made sure that without the @task decorator everything is working as expected.
I get the same results with and without "vpc_config".
This seems to be closely connected/identical to https://github.com/Miserlou/Zappa/issues/2127 .
I played around with excluding/including newer versions of boto3/botocore, pyopenssl, cryptography... no changes

johannesjasper commented 3 years ago

It seems to me that within the lambda running the WSGI app it tries to invoke itself (the same lambda) but cannot validate the SSL certificates of the AWS API.

Can somebody verify this very basic understanding?

johannesjasper commented 3 years ago

I updated the original post with a minimal example.

johannesjasper commented 3 years ago

While working on a workaround I discovered that ALL commuinication to AWS APIs is problematic. I tried to 'manually' (using boto3) send a message to SQS, with the same SSL issue

SSL validation failed for https://eu-central-1.queue.amazonaws.com/ [Errno 2] No such file or directory

The problem seems to lie somewhere in the base image used for Python Lambdas... :thinking:

johannesjasper commented 3 years ago

I used zappa shell dev in order to see if I can use boto3 inside my lambda

(Pdb) import boto3
(Pdb) s3 = boto3.resource('s3')
(Pdb) [bucket for bucket in s3.buckets.all()]
# --> [... lists all my buckets]

If my assumption that the execution environment was lacking some certificates was right, this should not have worked, right?

johannesjasper commented 3 years ago

Closing as duplicate of https://github.com/Miserlou/Zappa/issues/2127. The suggested workaround mitigated my issue https://stackoverflow.com/a/63642707/1821901.

Miserlou / Zappa