RSA_get0_crt_params: symbol not found

[guyzyl]

I built a Docker based on the ones here and baked in my own config (but replacing the config is the only change). I'm trying to run tit on GCP Run (with GCS) and receiving the following error when trying to log in:

Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/pyramid/router.py", line 277, in default_execution_policy return router.invoke_request(request) File "/usr/local/lib/python3.6/site-packages/pyramid/router.py", line 252, in invoke_request request._process_response_callbacks(response) File "/usr/local/lib/python3.6/site-packages/pyramid/request.py", line 83, in _process_response_callbacks callback(self, response) File "/usr/local/lib/python3.6/site-packages/pyramid_beaker/__init__.py", line 30, in session_callback self.persist() File "/usr/local/lib/python3.6/site-packages/beaker/session.py", line 875, in persist self._session().save() File "/usr/local/lib/python3.6/site-packages/beaker/session.py", line 723, in save self._create_cookie() File "/usr/local/lib/python3.6/site-packages/beaker/session.py", line 737, in _create_cookie val = self._encrypt_data() File "/usr/local/lib/python3.6/site-packages/beaker/session.py", line 379, in _encrypt_data return nonce + b64encode(self.crypto_module.aesEncrypt(data, encrypt_key)) File "/usr/local/lib/python3.6/site-packages/beaker/crypto/pyca_cryptography.py", line 21, in aesEncrypt backend=default_backend() File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend from cryptography.hazmat.backends.openssl.backend import backend File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module> from cryptography.hazmat.backends.openssl.backend import backend File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 75, in <module> from cryptography.hazmat.bindings.openssl import binding File "/usr/local/lib/python3.6/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 16, in <module> from cryptography.hazmat.bindings._openssl import ffi, lib ImportError: Error relocating /usr/local/lib/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: RSA_get0_crt_params: symbol not found

Tried using the following images: latest, latest-alpine, 1.1.1-py3-alpine

[stevearc]

I've pushed up a change to the container (should be in pypicloud:latest-alpine as soon as the build finishes). This was caused by some terrible OpenSSL binding issue that Alpine frequently runs into. I couldn't get it working with alpine 3.7, so I upgraded the container to depend on python3.8-alpine3.12 and that seems to have fixed the issue.

[guyzyl]

Thanks for the quick response! I'm running into a new issue with the new build: (If you want me to open a new issue for this let me know)

[stevearc]

I pushed up a change that I think might help, but I wasn't able to reproduce this error. Can you try again and, if the issue persists, give me some more information about your configuration?

[guyzyl]

Unfortunately I still receive the same error. The funny thing is that the error started showing up only after the first fix you pushed. A little about the deployment environment, I'm running the container on Google Cloud Run (Google's serverless Docker service). Nothing else special about the environment, using a config file generated using the make-config flag. Here's the full log of the container runtime, hopefully it will give some insight:

2020-08-03 07:27:15.828 GMT - [uWSGI] getting INI configuration from /etc/pypicloud/config.ini
2020-08-03 07:27:15.834 GMT - *** Starting uWSGI 2.0.19.1 (64bit) on [Mon Aug 3 07:27:15 2020] ***
2020-08-03 07:27:15.834 GMT - compiled with version: 9.3.0 on 31 July 2020 15:52:01
2020-08-03 07:27:15.834 GMT - os: Linux-4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016
2020-08-03 07:27:15.834 GMT - nodename: localhost
2020-08-03 07:27:15.834 GMT - machine: x86_64
2020-08-03 07:27:15.834 GMT - clock source: unix
2020-08-03 07:27:15.834 GMT - detected number of CPU cores: 2
2020-08-03 07:27:15.834 GMT - current working directory: /app
2020-08-03 07:27:15.834 GMT - detected binary path: /usr/local/bin/uwsgi
2020-08-03 07:27:15.834 GMT - !!! no internal routing support, rebuild with pcre support !!!
2020-08-03 07:27:15.834 GMT - your processes number limit is 1048576
2020-08-03 07:27:15.834 GMT - your memory page size is 4096 bytes
2020-08-03 07:27:15.834 GMT - detected max file descriptor number: 25000
2020-08-03 07:27:15.834 GMT - lock engine: pthread robust mutexes
2020-08-03 07:27:15.834 GMT - unable to make the mutex 'robust'
2020-08-03 07:27:16.118 GMT - Container called exit(1).

[stevearc]

Unfortunately I'm still unable to reproduce this issue. When I run the container I get the following output:

[uWSGI] getting INI configuration from /etc/pypicloud/config.ini
*** Starting uWSGI 2.0.19.1 (64bit) on [Wed Aug  5 04:34:45 2020] ***
compiled with version: 9.3.0 on 05 August 2020 04:33:33
os: Linux-5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020
nodename: 1abdb22af37c
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /app
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on 0.0.0.0:8080 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:39145 (port auto-assigned) fd 3
Python version: 3.8.5 (default, Jul 21 2020, 18:15:54)  [GCC 9.3.0]
Python main interpreter initialized at 0x5629b40caf40
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 15 seconds
mapped 1531320 bytes (1495 KB) for 20 cores
*** Operational MODE: preforking ***

Since I can't reproduce it, I wonder if the difference is in the kernel? I do seem to be on a newer version, but I hope that's not the issue. I don't have any great ideas, but I can keep throwing things at the wall. I pushed another change to the master branch that will force installing uwsgi from source instead of a binary. Hopefully building it from source will help it find whatever it's missing.

[guyzyl]

Still getting the same issue :( Would it be possible for you to try it out in Google Cloud Run?

[stevearc]

I've never used Google Cloud before and to be honest I really don't want to spend my time ramping up on another cloud platform that I'm never going to use again. The initial issue here was solved, and it seems like the only remaining problem is something that's platform-specific. If you want to mess with the Dockerfile yourself and find a workaround, I'd be happy to look at the pull request!

[guyzyl]

@stevearc that makes sense, thanks a lot for the help up until now!

[guyzyl]

It turns out that Google Cloud Run runs the containers with gVisor (see here), I'm assuming this is what's causing the mutex issues.

[guyzyl]

After looking a bit in both uWSGI and gVisor code, it seems the problem arose from uWSGI trying to acquire the robust mutex (here) which isn't possible when running on gVisor. Fixed this by setting enable-threads = false in the [uwsgi] section of the config which causes it not to try and aquire the mutex. Closing the issue 🙂